Wow ...

The company that produces your favourite security researchers' favourite tools has been acquired by Google. You, dear reader, have every right to be surprised; we ourselves are still recovering from the happy surprise.

"In mathematics, you don't understand things. You just get used to them."

-- John von Neumann

Old Jonny was right, but you might substitute "mathematics" with "life" in the above.

zynamics was never designed to be acquired. To be quite honest, zynamics wasn't designed at all -- it mostly just "happened". We never had a plan outside of "build the tools that we want to have, others will then want to have them too". We never took venture capital, and the only business plan I ever created was a half-baked attempt made with wizard software. It was never updated. The fact that we exceeded the forecasts was mostly due to me being an overly pessimistic planner.

Calling our structure engineering-centric would be an understatement; our everything-to-engineering ratio is roughly 7% (if I may still count myself as an engineer).

As we grew, the problems that we wanted to solve grew too -- at a much faster rate than our resources to solve them. The result was that I spent more and more time running around managing, doing sales, and acquiring resources so that my team could do the technical work that I love and am good at. I wanted the chance to focus on technical issues again.

So at some point in this process we started talking with Google, much to our own surprise. We had not anticipated this -- we are not web-centric, we are far away from their core business: At first glance the acquisition seemed like a strange choice for both sides.

Yes, we have excellent technology and a core of great engineers; we were just surprised about the fact that Google would be interested. It was certainly not an obvious pick.

Then again, Google shares an engineering-centric culture, and has just the resources we're lacking.

"The purpose of computing is insight, not numbers."

-- Richard Hamming

A friend of mine (rightfully) mocked me for trying to perform serious computation on something that strongly resembles a pocket calculator. According to him, there's roughly one and a half computers in this world -- and Google happens to have one of them. I tend to concur.

So, as of today, I can say that the entire zynamics team has joined Google. I am looking forward to tackling problems with the resources that Google can provide.

While there will be some changes, our products are not going away - on the contrary !

"Et si tu me demandes quel est donc ce 'propos' que je poursuis a longeur a mille pages, je repondrai: c'est de faire le recit, et par la-meme la decouverte, de l'aventure interieure qu'a ete et qu'est ma vie."

-- Alexandre Grothendiek

Running zynamics with my team was an exciting experience, but I have no doubt that the future will be every bit as exciting.

Trainings class with SP and me at CSW !

Hey all,

SP and me will be teaching a trainings class this year at CanSecWest. If you have some background in reverse engineering and want to

• become a more efficient reverse engineer
• become a more efficient bug hunter
• become better at understanding stuff like Acrobat's JScript Engine

this class is for you. We will teach you stuff including but not limited to:

• Quickly find where the interesting parts of the executable are: Who is parsing user input ? Who is responsible for the crypto ?
• Save time: Identify what open-source libraries are statically linked into the executable. Why audit binary when you can read source ?
  
• Want to understand what Acrobat is doing ? Or most C++ programs nowadays ? Generate UML diagrams from binaries, showing you all the classes and their hierarchy.

Anyhow, follow this link if you are interested. I think it's going to be a blast.

Cheers,
Halvar

Tax evasion and welfare fraud

Hey all,

now that all the technical stuff is going to the zynamics company blog , I will have some room here for writing about other topics. Beware: Politics might be involved, or just general rants.

Tonight I will write a little bit about tax evasion and welfare fraud. I somehow wound up in a discussion about the topic, and the end result was that I spent 20 minutes doing a bit of research on the topic.

Background: The German government was offered a CD containing data of people that have moved money into swiss bank accounts, presumably to evade taxes. The person offering the CD claims that it contains almost exclusively data of tax evaders, and demands a fee of 2.5 million EU to provide the CD to German authorities.

This situation has spawned a debate about the legality of the situation: Is it "right" for the German government to buy data that was obtained in a presumably illicit fashion ? (I intentionally avoid "illegal" here -- the person that obtained the data might be in breach of contract with his employer, but it is unclear whether he broke any criminal laws)

Clearly, it is a tricky question - but the difficulty of this question is not the topic of this blog post.

Recently, a German politician (who, ironically, was repeatedly involved in corruption affairs, most notably in the CDU-party-donations affair) by the name of "Roland Koch" argued that welfare fraud is a serious problem in Germany, and that 15% of all welware recipients do not actually want to work. He argued for annuling benefits of these 15% in a large conservative newspaper (the FAZ).

So in todays discussion, the question came up: What is actually the "bigger" crime (in terms of financial damage): Tax evasion of welfare fraud ?

It is relatively straightforward to calculate the cost of welfare fraud: Germany spent 21.7 billion EU in 2008 on the "Hartz-4" system. This includes administrative overhead. Assuming that Mr. Kochs claim has merit, and assuming that overhead is also inflated due to fraud, ~3.3 billion EU are lost annually to welfare fraud.

It is much more difficult to calculate the cost of tax evasion. There are many numbers that are difficult to justify, and most appear to be made up arbitrarily. The only halfways reliable number I could find was from this article:

The amount of money generated from tax investigations that followed evasion was ~1.6 billion EU in 2004. Inflation-adjusted to 2008 at 2% inflation, this ends up being ~1.73 billion.

This implies something rather interesting:

1. Assuming that every third tax evader is caught (which I deem more realistic, just by gut feeling, e.g. without any scientific base), tax evasion is already a much bigger problem than welfare fraud.

The question of course is: What is the actual rate of tax evasion to "getting caught" ?

The new, shiny, reverse-engineering-centric zynamics blog !

Hey all,

for all those that have almost gotten sick of me posting only rarely on this blog:

We have a shiny new reverse-engineering-centric blog up on http://blog.zynamics.com ! :)

The entire team will post RE-related issues there, so I think it'll be a rather good read :)

Interesting Blog Posts

Hey all,

so while you, dear reader, are still waiting on me finally fullfilling my promise about blogging more, I have some interesting links for you :)

Vincenzo Iozzo has been blogging about some cool stuff he has been doing using NaviPython, REIL, MonoREIL etc. recently, and you can read about it here:

http://viozzo.wordpress.com/2009/12/11/scripting-with-binnavi-cyclomatic-complexity/
http://viozzo.wordpress.com/2009/12/18/finding-interesting-loops-using-monoreil/

Cheers,
Halvar

Clarification to the previous post

Hey all,

I thought I need to clarify something about the previous post: I was trying to explain the fact that people reacted harshly to the hint that a new standard is being drafted, without knowing anything about it. So I talked about the historical perspective on the old OIS draft, and what my thoughts about it were, and what I think the reasons are that researchers usually do not bother with these things much.

This was not meant as a reaction to the ISO standard under discussion. Katie Moussouris clarifies a lot of important points here -- and what she writes is completely sensible.

Anyhow, enough of this :-). The upside of the entire discussion: I really like the pun in the above link. Yes, I know that I have a weird sense of humor.

Why are most researchers not a fan of standards on "responsible disclosure"

I usually try to stay away from the politics of vulnerability disclosure, mostly because I think (to paraphrase Feynman) that politics of vulnerability disclosure are as useful to the vulnerability researcher as ornithology is to birds.

But it seems that the entire discussion is not going away. The intensity of the reactions to k8em0's twitter post might be partially explained by the history of this all. I'll try to refresh what I remember:

A lot of the older vulnerability researchers remember the ghastly OIS attempt at forcing a standard written by a bunch of non-researchers down the throats of the research community. From the outside, it looked mostly like an attempt to kiss up to some vendors that were spending a lot of money on security review during that time.

I might be stepping on some people's toes, but to me it looked like a high-school class where the dimmest students drew up guidelines on how smart students "should" behave, and gave that to the teacher in order to earn brownie points - including clauses like 'not contradicting the teacher'.

Unfortunately, most of the research community prefers to do work instead of discussing with people that have little interesting to say about how the researchers should work. The result of this is that researchers were rarely ever involved in the entire discussion. Not for lack of opportunity, but mostly lack of interest -- if I can actually go and surf, why would I discuss with a bunch of people sitting in an office about the right way to come back to the beach ?

The entire discussion has always been somewhat phony. The entire "responsible/irresponsible" angle is sligthly fraudulent. The way I see it is the following:

1. It is acceptable for AV companies to charge for signatures, which are in essence "information about malware"
2. It is acceptable for AV companies to not publish, nor provide, malware to other parties, or to charge for it
3. It is acceptable for software vendors to charge so I can use their software. It is also acceptable for them to charge more so that I can read their source code.
4. Why again should a researcher be obliged to provide information to vendors free of charge again ?
5. If anyone argues it's "responsible" to make everyone safer, I say: I'll give all my bugs to all vendors the same day that all security companies of the world provide free licenses for everyone for their software.
   

But well. Honestly, I am not sure whether I should post this. I do not really feel like spending too much time discussing this. But perhaps that's part of the problem...

Low blogging frequency

Hey all,

first of all, I seriously have to apologize for the low frequency of blog posts nowadays. We have been doing a bunch of interesting things at work that I will post about shortly. Amongst the
things on my "to-post" list are:

• Rants on our experiences distributing VxClass
• A method to perform exact directed graph comparison in O(1) (with some caveats ;) -- we have been sitting on this for a year or three, but were caught up in other things so writing it up fell by the wayside
• Automated generation of byte signatures from the VxClass results
  

Anyhow, expect a higher blogging frequency from this blog in the next weeks. I will restrict my use of twitter for this.

Looking for Memoryze dumps of malware

Hey all,

I am looking for Memoryze dumps of various pieces of malware -- the more the merrier. Does anyone here have some ?

Cheers,
Halvar

Adventures in choosing what to vote

accidental posting, not ready yet

Restaurant Review: "Le Surfing", Les Estagnots

Hey all,

I know it is a bit odd to post restaurant reviews on this RE-centric blog, but ... err ... I'll do it anyhow.

So I am currently on my vacation (yay) in southwestern france -- close to Seignosse/Hossegor. On my search for wifi, I ran into a place called "Le Surfing", which is a small ... well ... surf-bar (?) at the beach "Les Estagnots".

The place is definitely pretty damn relaxed -- it seems that a bunch of folks that want to be at the beach decided the easiest way to finance it is through this bar. When I first walked in, I was greeted with "hey, there's a customer, we need someone behind the bar".

So tonight I decided to eat here. After trying to order the Thai-style beef salad, the waitress convinced me that it's saturday, and on saturdays they have sushi, and I absolutely have to eat their sushi. So I ordered their sushi.

And I didn't regret it. It's rather unorthodox -- e.g. it features rice, raw fish, nori, wasabi, and soy sauce, but I wouldn't describe it as traditional sushi. Not that this is bad: I had a stellar roll that consisted of crab meat, a mild goat cream cheese, cucumer, and bell pepper. There was some tuna, fresh crevettes, an onion-heavy-salad, and various other things.

All in all, it was really good. Seriously.

Open position :-)

Hey all,

we have an open full-time position in our office in Bochum, Germany -- in essence, a vuln-dev-using-BinDiff-and-BinNavi-position.

What are the tasks in this job ?

• Use BinNavi and BinDiff to find bugs in software & write exploits
• Test BinNavi/BinDiff features. Test them some more.
  
• Provide feedback to the developers about the tools
• Write up your results
• (Potentially) train clients in the use of the tools

We're looking for someone that has some experience in RE and vuln-dev.

What we can offer:

• A no-bullshit environment
• Small team, good mood in the office :-)
• Direct influence over the development of the tools
• Probably more than that, but it is late and I am tired
  

If you're interested, please send email to halvar.flake@zynamics.com

Klartext please !

Hey all,

I won't comment too much here right now (I am kinda busy with work), but imaginably in relation to my last blog post:

http://www.microsoft.com/technet/security/bulletin/ms09-jul-ans.mspx

Now my favourite quote from http://blogs.technet.com/msrc/ :

"While we can’t go into specifics about the issue prior to release, we can say that the Visual Studio bulletin will address an issue that can affect certain types of applications."

I would be tempted to say that "if you can't say anything useful, say nothing at all", but I am often guilty of writing useless things, so I won't.

The above formulation inspired me though:

• "While I have no clue what this out-of-band patch is all about, it will address an issue that has to do with a programming mistake."
• "While going into the specifics would be dangerous to the safety of the world, the issue that will be addressed is an issue that can affect computers."
  

Sigh. Newspeak.

There is a German expression called "Klartext reden" (literally translated 'to speak clear text'). If you know a german, grab him, and ask him to explain the meaning of the word to you. I wish there was more of this around.

So my 2 cents of speculation:

• MS will patch a bunch of libraries (the ATL ?) in Visual Studio
  
• A "certain type" of application is affected and will need recompiling. The msvidctl thingie was an ActiveX component. Anybody wants to add 2 and 2 together ?

I really need to get back to work though.

Poking around MSVIDCTL.DLL

Hey all,

I have to admit that I did not follow the msvidctl.dll situation all that closely, except for my short tweet that this bug was apparently reported more than a year ago. Yesterday, my old friend Dennis Elser piqued my interest: He had isolated the bug down to a function called ATL::CComVariant::ReadFromStream. He had determined that the function in question made a rather strange mistake: Instead of passing a pointer to a data buffer to IStream::Read, it took the address of a (small) local variable, and passes this address as output buffer to IStream::Read, along with a length read from the stream previously.

Somebody clearly got confused.

So Dennis and me sat down tonight and did a bit of digging around and tried to clarify what was going on. So we dug in a bit deeper, and ended up with the following understanding of the situation:

1. If the stuff that is supposed to be deserialized does not contain the proper value in the first 2 bytes, 8 more bytes are read, and SafeArrayCreate is used to create a new array with a 4-byte size obtained from these bytes.
2. A pointer to the allocated memory is obtained by ways of calling SafeArrayAccessData. This function places a pointer to the memory in question into a memory cell pointed to by it's second argument.
   
3. Instead of passing the CONTENTS of the memory cell pointed to by the second argument to IStream::Read, the code in question passes the address of this variable in. This happens to be the re-used memory for the first argument of ReadFromStream, hence it is on the stack. Memory corruption hilarity ensues.

This is a cute little bug. First of all, it is a beautiful example of a single excess "&" in the source code. But what is most amusing about this bug is the centrality of it -- it is, after all, in a method of a class from the ATL.

Everybody loves bugs in libraries. Few things fill my heart with quite as much amusement as bugs in heavily-used, statically-linked libraries. OpenSSL (and libeay) was good for many laughs in the past, zlib was a favourite for a long time, too.

So what we have here is a bug in a component that is used fairly widely, and that has the property of being statically linked (yay templates !).

Now, a quick search of my harddisk turned out that a lot of third-party components (flash) contain the same function -- but in old and non-vulnerable versions (for an extra dash of irony, the function used to be safe before all this SafeArray-stuff was added). Only a small fraction of the files that use the ATL and contain this function appear to contain it in a vulnerable version.

We ended up building a really naive / stupid / false-negative-and-false-positive-prone regexp to scan for the vulnerable basic block. This is of course going to fail if anyone has tweaked their optimization settings etc., but it would still be interesting to find out how many files contain this "trivial" byte string.

So I searched my windows directory for the following regexp pattern:

pattern = "\x8B\x07\x6A\x00\xFF\x75\x2E\x8D\x4D\x2E\x51\x57\xFF\x50\x0C\x53"
r = re.compile( pattern, re.DOTALL )

There were a few files in which this pattern was found (XP):

Found pattern in file c:\Windows\system32\ieframe.dll
Found pattern in file c:\Windows\system32\mstscax.dll
Found pattern in file c:\Windows\system32\msvidctl.dll
Found pattern in file c:\Windows\system32\wmp.dll
Found pattern in file c:\Windows\system32\wmpdxm.dll

Dennis searched for the same pattern on his disk (Vista) and found:

c:\windows\system32\cic.dll
c:\windows\system32\comsnap.dll
c:\windows\system32\comsvcs.dll
c:\windows\system32\ieframe.dll
c:\windows\system32\mmcndmgr.dll
c:\windows\system32\mstscax.dll
c:\windows\system32\MSVidCtl.dll
c:\windows\system32\puiobj.dll
c:\windows\system32\rdpencom.dll
c:\windows\system32\shdocvw.dll
c:\windows\system32\wiaaut.dll
c:\windows\system32\wmp.dll
c:\windows\system32\wmpdxm.dll

Why the difference ? Well, amusingly, shdocvw.dll on my XP machine doesn't have this SafeArray-stuff in it yet -- it is probably compiled using an older, not vulnerable variant of the ATL -- wheras Dennis variant of the same DLL is much newer, compiled with the 'broken' variant of the ATL.

So, where does this leave us ?

1. The bug is actually much "deeper" than most people realize.
2. The killbit-fix is clearly insufficient, as there are bound to be many other ways of triggering the issue.
   
3. The bug might have weaseled it's way into third-party components, IF anyone outside of Microsoft had access to the broken ATL versions.
   
4. If this has happened, MS might have accidentally introduced security vulnerabilities into third-party products.
5. Depending on the optimization settings applied to the executables, it might require a bit of an effort to find out whether a vulnerable or non-vulnerable version of the code is present.
6. There might be a lot of recompiling next week.
7. IF this has gotten into third-party-products, I would bet that only a tiny fraction of software vendors will push out proper/timely updates.

It just seems that spending time to improve BinDiffs ability to find statically linked libraries might have been worth it :-)

Anyhow, I really need to get to sleep -- I have a train to catch in a bit more than 4 hours.

A lot of credit for this post has to go to Dennis Elser -- he did most of the hard work before we sat down.

Las Vegas / Blackhat / DefCon

Hey all,

the annual Blackhat / DefCon thing is coming up again, and I just wanted to announce here that I will be in Las Vegas from the 24th of July to the 2nd of August. I will be busy teaching classes until the 28th of July, but thereafter I am mostly free (e.g. roaming the conference). If you wish to schedule a demo for any of our products (BinDiff, BinNavi, VxClass) or just generally have a chat about all things related to reverse engineering, vulnerability analysis, malware classification, deobfuscation, math or business, do not hesitate to contact me beforehand :-)

Cheers,
Halvar

Strange Cellphone Behavior

Hey all,

I know this blog post is a bit weird, but I reckon I'd share this: For some reason that is quite unknown to me, my cellphones have a habit of developing strange behaviors. I used to use a Nokia N73, which developed the following habit:

When in foreign time zones (Japan, Norway, USA) the phone would send more-or-less random old text messages to more-or-less random people from my address book. There would be a merry mix & match between the two, leading to more than one amusing misunderstanding that needed clearing up.

Then, at some point last fall, I switched to the silly shiny Apple telephony device (perhaps people do better QA on their backdoors on that platform). For a few months, the problems went away.
This changed last week -- now, when I send text messages to certain numbers, the phone seems to send a more-or-less random old text messages that has already been sent to the same number along with the message. This is a bit nicer (as it will not mix & match), but still annoying.

So .. uhm ... I am trying to come up with plausible explanations for this behavior. Can anyone offer one ? My total-guess-in-the-dark ideas would be:

1. Current behavior is caused by international text message routing weirdness -- e.g. text messages I sent a few days ago in the US get duplicated for some reason and re-sent
2. Both current and N73 behavior is triggered by shoddy QA on lawful intercept systems
3. Both current and N73 behavior is triggered by shoddy QA on the side of the parties that backdoor my phones

Now, I don't know if anyone else has ever suffered from this, or if there is a perfectly valid and proper explanation, or if there is an easy way to do diagnostics, but:

1. If you backdoor my phone, fix your software. Kthx.
   
2. If you write LI software, fix your software. Kthx.
   
3. If there are multiple people backdooring my phone, please test for interoperability between your tools.
   

So, any other theories on what might be going on ?

Reverse Engineering / Bug hunting trainings in Amsterdam

Hey all,

I haven't given a reverse engineering trainings class in Amsterdam for a few years, but this year is different :-) -- I will be at BH Amsterdam, and there are still seats open in the trainings class for April 14th and 15th.

What will be done in the course ? Well, for one thing, we'll go bug-hunting in some interesting piece of code. Furthermore, we'll talk quite a bit about C++ and it's effects in the binary. We'll do a fair bit of differential debugging, some more bug-hunting, and a lot of IDA automation. Questions like

• given a C++ executable, how do I recover an inheritance diagram of the classes ?
• given a big and ugly executable, how do I find the interesting places to focus on ?
• how do I make sure IDAPython and NaviPython make my life easier ?

will be treated thoroughly.

So, if you still have some trainings/travel budget left in spite of the crisis, you can find more
details here.

Diffing x86 vs ARM code

I posted a while ago about the new DiffDeluxe comparison engine, and that we'd release it in Q1 2009. Well, we're almost there, the engine is now in beta. If you are a BinDiff user and wish to give the new engine a try, send mail to info@zynamics.com :-)

I mentioned in my last post on the topic that DiffDeluxe was designed to facilitate symbol porting, and to allow comparisons between executables that are "far away" from each other.

In the last post I wrote about Mozilla JS engine vs. Acrobat EScript.dll. Today I am going to try something slightly crazier: In order to evaluate how well these matching algorithms work, we will be diffing an executable that was compiled for ARM against a very similar executable compiled for x86.

My coworker Vincenzo is a big fan of all things OSX, and he brought up the idea of comparing x86 and ARM versions of the OSX dynamic loader -- namely the disassembly of dyld on the iphone against the disassembly of dyld on OSX.

Now, the first voices are going to yell: "You have names for all functions, BinDiffing is easy then!". Well, true, but we will run DiffDeluxe without taking the names into account, and then just using the names to validate the results.

The two executables have 704 (x86) and 618 (ARM) functions respectively. Without name
matching, we match 345 functions. Inspecting the symbols, we see that we have matched
160 of these functions in full accordance with the symbols. Let's have a look at some of the details:

Cute, eh ? Let's look at some more...

It is almost surprising how far one can get without actually looking at the instruction semantics.

If we take the names into account, matching functions becomes easy, but matching basic blocks properly ends up the difficulty. With name matching enabled, DiffDeluxe matches 3809 basic blocks, out of 7904 respective 5196.

So to summarize: The structural comparison is sufficiently strong to yield some useful results even accross two different CPUs. While there is still (a good amount) of room for improvement, I am quite happy with these results so far :-)

So, if you want to beta, and you already use BinDiff, drop us a line !

Washington DC, Trainings, Demos :-)

Hey all,

I will be in Washington DC from the 16th to the 20th of February. Amongst other things, I will be teaching a course at Blackhat DC. The economic crisis is clearly hitting -- e.g. there are still seats available. We will also get around to using some of the nice features of BinNavi v2 in class, which I am looking forwards to.

Now, aside from the course: If you are in the DC area and interested in a product demo for BinDiff (and the upcoming DiffDeluxe), BinNavi v2 (including REIL), or the latest VxClass (now available as service and virtual appliance), do not hesitate to drop a line to info@zynamics.com :-)

Correction: Clam *does* have some unpacking support

Correction of my last post: It appears that Clam has *some* unpacking support. It is not as comprehensive as some of us would like, but progress is being made :-)