Research on the given file-type/title, and analysis of a files header are pretty good starting points.
This topic might be a good starting point. If you're tackling any modern title post-2020, the vast majority of titles aren't going to be very interesting, zlib/deflate/lz4/lz4hc/lzma/lzo/zstd/oodle are going to be the algorithms you'll find most of the time, most of the time xtool can handle these out-of-the-box, or with a plugin. There are some exceptions but, at least for the time being Godot/Unity/Unreal are the engines most titles opt to use. Thankfully some indies and larger studios/publishers still use/develop their own in-house engines which give you something more interesting to work with.
It's the variants that will give you some trouble (i.e. the 2009 avatar title, The Surge series (LZ4 implementation isn't custom, just a bit funky), Far Cry 2, State of Decay 2, Sleeping Dogs, to name a few off the top of my head) these will require you to either create your own tools/solution or forgo processing the streams altogether.