>NET, Julia and uv's tar-codec - These implementations reject the tarball. I believe this is the best thing you can do, even if it's technically not POSIX-compliant.
While this is the safest choice, it is the least helpful, I would always prefer to get something out of an archive then nothing. Damage/irregularity to an archive shouldn't prevent data inspection, just a warning and a way to fall back with caveats for what can be extracted.
Iāve been working on an open source tool for creating SBOMs from binaries/file systems. Automated extraction of archives like zip/tar files present interesting challenges - for a while Iāve known about OS filesystem formatting gotchas like archives from a Linux-based system with symlinks or case colliding names not extracting correctly on NTFS (and HFS+ for the latter case).
But the ambiguity of this tar filesā¦ I guess the ācorrectā extraction for what Iām doing would require figuring out what the system that is processing the tar files used and copy that behavior, which is non-trivial if you donāt know what an embedded device is using behind the scenes.
Astral blogged about their new tar parser for uv recently, see https://astral.sh/blog/tar-codec. I've included it in my work-in-progress test suite https://stoppels.ch/tar/ which shows it's by far the strictest tar library out there.
For your use case it would make sense to pick the strictest tar parser, such that you can error or warn on most ambiguities. In practice these ambiguities are rare.
I feel like the most faithful extraction would dictate that any field in the original header should take precedence over the PaxHeader's iff the PaxHeader field's value would've been representable in the original field. Anyone have thoughts?
>NET, Julia and uv's tar-codec - These implementations reject the tarball. I believe this is the best thing you can do, even if it's technically not POSIX-compliant.
While this is the safest choice, it is the least helpful, I would always prefer to get something out of an archive then nothing. Damage/irregularity to an archive shouldn't prevent data inspection, just a warning and a way to fall back with caveats for what can be extracted.
Iāve been working on an open source tool for creating SBOMs from binaries/file systems. Automated extraction of archives like zip/tar files present interesting challenges - for a while Iāve known about OS filesystem formatting gotchas like archives from a Linux-based system with symlinks or case colliding names not extracting correctly on NTFS (and HFS+ for the latter case).
But the ambiguity of this tar filesā¦ I guess the ācorrectā extraction for what Iām doing would require figuring out what the system that is processing the tar files used and copy that behavior, which is non-trivial if you donāt know what an embedded device is using behind the scenes.
Astral blogged about their new tar parser for uv recently, see https://astral.sh/blog/tar-codec. I've included it in my work-in-progress test suite https://stoppels.ch/tar/ which shows it's by far the strictest tar library out there.
For your use case it would make sense to pick the strictest tar parser, such that you can error or warn on most ambiguities. In practice these ambiguities are rare.
Disregarding erroring as an option-
I feel like the most faithful extraction would dictate that any field in the original header should take precedence over the PaxHeader's iff the PaxHeader field's value would've been representable in the original field. Anyone have thoughts?
This guy, what a hero. He even included a little graphic to show how this "hardest tarball" looks like.
At the bottom, there's a link to much more edge cases.
It reminds me of this beauty: "Parsing JSON is a Minefield" https://news.ycombinator.com/item?id=16897061