Changing header of executable

[ptsz]

Hi,

I was wondering if someone knew how I could modify the header of an executable to NOT open a console window? After the executable has been compiled. I know you usually do that during compiling but.

It is my current understanding that you can modify the header of the exe file to not open a console window behind it. I hope that correct.

Hopefully someone can help me?


Thanks.

[TippeX]

change the os subtype setting in the pe header from win cui to win gui... simple patch

[ptsz]

Thanks for the response.

Could you also tell me how to do that, which tools and stuff to use plus well some notes on how to do it. :)

I have PE explorer, don't know if thats the one to use. Obviously I have a hex editor.

Using PE explorer. When I open it up, the header.

Subsystem 0003h Win32 Console

This is what you mean I guess? Close I hope at least. :)


I am new to it, but happy to learn how to do it.

[TippeX]

yup thats it. .
if i remember right gui is 02 so change that, job done

[ptsz]

OK. And how do I change it? Because with PE explorer I can't as it seems. And with neo hex editor I didn't find the 003h (because I couldn't search for an "h").

Do you know another software I should be using?

And can I do this with an app without a GUI as well?


EDIT:

I found it. Thank you very much for the help. So glad I could FINALLY get it to work. :)

[TippeX]

h = hex, can also be 0x
its convention...

changing the subsystem isnt always a good idea, especially if it outputs to the console

[Joe Forster/STA]

Yup, if the software expects to be running as console/GUI - and what software wouldn't?! - then changing its subsystem will make it go completely crazy. I just tried it with one of my own programs, originally console and hacked into GUI: after processing its input and sending the results into its output (redirected into a file so that I can see it really did something), it keeps running as if it is unable to realize that its input has ended.

(For hacking executables, Hacker's View is probably your best choice.)

[ptsz]

Yea I wouldn't expect user input to work anymore probably.

But for some that don't use or need it, it would be great. Plus executables compiled from other languages that may not have a console remover in the compiler as option.

[TippeX]

not true..if its cui..its there for a reason..
example.. try looking at some dlls in system32

[ptsz]

OK, which one in particular?

I ran the app though process explorer but it uses a lot, so I didn't know which in particular you meant.

PS: Also on another note, where should/can I ask about if anyone knows anything about LZSS compression? Which forum topic would that be?

[TippeX]

quite a few of the system dlls are compiled as cui... have a look :)
im guessing this question is driven from something like gta iv or maybe la noire that pops up a useless cui window?

but just changing it can cause problems as some compilers / languages have subtle differences to the winmain / dllmain / main depending on how it was compiled

whats the compression question?

[ptsz]

Oh no, it doesn't derive from game exe's. More from other languages which would not have the option to compile it without a console. Or just exe I would like to change or make, but lets say not having the option to remove the console.

You can also do it by running a winapi program calling the designated app by using "execlp("notepad.exe", 0);" but than you would of course need a second app to run the initial one without a console. :)
But it would have the same effect. In general the apps would be C/C++ coded (except if in another language).

I noticed some dlls compiled as cui yes.

The compression question would be. I have a game which uses LZSS compression for most of its content and I would like to hack it. It has already been hacked so I have the AES encryption key and you can extract it. The problem is that every file above 200k is corrupt, completely. So you can't use them.
That is the problem. I use QuickBMS to extract it. The problem is with the quickbms extracting script but I am not qualified enough to fix it properly. I was hoping someone else would. :)
I asked the author if he would help but he didn't want to.

I have the files, 1 win_000.nfs file 8mb and 1 win_000.full 4gb ready and uploaded (including both pack/unpack scripts). But I need someone who can help. Basically someone who knows something about LZSS compression to get it right.

[Joe Forster/STA]

You can run a console program without a console using start /b <program>. But that works from the command line interface - %COMSPEC% - only as it's an internal command. It won't work from a GUI program which is why I wrote two versions of the "Minimize2Tray" programs - console and GUI -, to be released some time...

[TippeX]

yeh, i always wondered why a hybrid type didnt exist.
ie: run from within console then console mode, otherwise gui...would be nice

also only realised "hackers view" as hiew (what i usually call it) - excellent program..i even got work to license a copy...author is a nice guy too

[Joe Forster/STA]

i always wondered why a hybrid type didnt exist.

Hmmm, interesting, never thought of that. Actually, the program, compiled as GUI, can find out what type its parent process is and if it's a console then open a new console or attach itself to the parent's console; there are simple API calls for that. But I know already one problem: launching a GUI program from the console will return at once without waiting for the program to initiailize/send some signal/exit (start /wait <program> helps, though) and there can be more...

[TippeX]

yup...AllocConsole etc...more overhead though.. maybe win10 or smth can go hybrid

[ptsz]

which is why I wrote two versions of the "Minimize2Tray" programs - console and GUI -, to be released some time...

I have been looking for source code how to minimize an app to the tray for a while. Unfortunately it has to be WinAPI but. Do you know or have a source code on how to do that by any chance?


Also what about my LZSS compression problem? :)

[Joe Forster/STA]

There are hundreds of "minimize to tray" programs out there but because 1) none did exactly what I wanted and 2) I've been in a C coding spree for years anyway ;) I wrote my own. Drop me an E-mail and I'll send to you the binary as well as the full source (MinGW/gcc 4.5+ needed for compilation). And, no, it's not trivial how to do it... :)

Sorry, I can't help you with your other problem; I'm not very familiar with compression beyond the basic knowledge and some hacking.

[ptsz]

Oh, thanks. I will do that. :)

I'll send it by PM if thats ok.

Glad you have it in C as well. :)

Also, how does it work? So you have/made an app to minimize another to the tray right? But obviously I could input the same code into the app I would like to minimize itself right?

[Joe Forster/STA]

Minimizing your own application into the tray is pretty easy; minimizing another application is complicated. ;)

(E-mail addresses on common mail providers are scrubbed in PM's as well as the public forum, to protect them against E-mail address harvesters that spammers use. Mine is protected well against spam anyway so send me an E-mail.)

[TippeX]

the compression thing sounds like the algo is going bad somewhere or there is some crypto on the buffer after x bytes, usually thats the cause...try using the algo to compress something else then decompress and compare

[ptsz]

I don't have the algo though, to test out on other things. I did do some searching but didn't find exactly what I wanted.

I just got something to test it, some sort of LZSS v0.0.1 tool, it works fine with files above 200k.

But in any case, here is the unpack script that it uses to unpack the files:


script temp removed


It is said to contain some errors (possibly?), maybe you can tell? I have the files ready in case you wanna give it a shot.

I haven't done any LZSS compression or such so far though, only tried to extract it for quite a while by now. :)

[links4all]

You can check the source of UPX of course for this.
I know another website which uses this technique and publiced some information about the headers, I thought it was http://www.exesecure.com, they do the same actually. Maybe you can recompile the program for convenience.