You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I vaguely recall exploring this concept and finding that tarfiles don’t supply the requisite interface because they’re not random access. I’m only 10% confident in that recollection, so worth exploring.
That is good to know. This isn't very high on my priority list, but I will try to explore when I have some time.
barneygale commented on Jan 1, 2022
barneygalemannequin
Mannequin
More actions
It's possible to do, but will be a little slow due to the nature of tar files. They're a big linked list of files, so you need to do a bunch of reads/seeks from the start to the end to enumerate all files.
I'd ask that we try to get bpo-24132 solved first. That would let us write:
I'd recommend not to block on bpo-24132. It's not obvious to me that subclassing would be valuable. It depends on how it's implemented, but in my experience, zipfile.Path doesn't and cannot implement the full interface of pathlib.Path. Instead zipfile.Path attempts to implement a protocol. At the time, the protocol was undefined, but now there exists importlib.resources.abc.Traversable (https://docs.python.org/3/library/importlib.html#importlib.abc.Traversable), the interface needed by importlib.resources. I'd honestly just create a standalone class, see if it can implement Traversable, and only then consider if it should implement a more complicated interface (such as something with symlink support or perhaps even later subclassing from pathlib.Path).
barneygale commented on Jan 2, 2022
barneygalemannequin
Mannequin
More actions
If you're only aiming for Traversable compatibility, sure.
The original bug description asks for something that's pathlib-compatible and similar to zipfile.Path, which goes beyond the Traversable interface in attempting to emulate pathlib.Path.
The pathlib.Path interface is a good one - I see no reason it can't apply to zip and tar archives in full. Methods of Path objects already raise NotImplementedError if operations aren't supported (e.g. creating symlinks)
Anyone implementing this should be aware that tar members don't have unique names.
A path should generally refer to the last member of the given name (which would overwrite previous ones when extracting -- unless the previous members do shenanigans with symlinks/directories, but API for accessing individual members isn't expected to take that into account).
TarFile API like getmember will take care of most of this, but not all: for example, a “listdir” operation should probably de-duplicate its result.
What should tarfile.TarPath do with symlinks in archives? In the PR above they're followed in the same circumstances as in pathlib.PosixPath, but that might not be desirable/safe default. We could add an initialiser argument that disabled following symlinks entirely - either omit them or represent them as regular files, somehow?
If the answer isn't clear then I'd like to incubate this in a PyPI project for a while first
@encukou I saw you were working on tarfile recently. Pinging in case this interests you!
IMO, symlinks to files that are in the archive should continue to work.
Perhaps relative symlinks to files within the archive root should work too, though these become dangerous if there are any directory symlinks pointing outside.
In general, if an archive contains directory symlinks, you can only get the final destination of a file by extracting the archive (or by doing a very elaborate dry run, for which tarfile isn't suited).
It's very much not clear-cut. A tarball is, essentially, a collection of procedural instructions that allow merging contents with an existing filesystem in ”interesting” ways, while TarPath would like to work with a final tree structure.
IMO, to answer questions like this, we first need a good conceptual model of what TarPath represents.
A PyPI project that handles the simple cases sounds like a good start.
Thank you, @encukou! I'll move the TarPath implementation in #106337 into an experimental PyPI package then, leaving just the (private) pathlib._VirtualPath in that PR.
we first need a conceptual model of what TarPath represents
FWIW, I won't make one any time soon -- I'm taking a break from tarfile after a few months of dealing with it.
But if you want to bounce ideas off me, I'm here.
Activity
FFY00 commentedon Oct 28, 2021
It would be helpful to have a pathlib-compatible object in tarfile, similarly to zipfile.Path.
jaraco commentedon Oct 28, 2021
I vaguely recall exploring this concept and finding that tarfiles don’t supply the requisite interface because they’re not random access. I’m only 10% confident in that recollection, so worth exploring.
FFY00 commentedon Oct 28, 2021
That is good to know. This isn't very high on my priority list, but I will try to explore when I have some time.
barneygale commentedon Jan 1, 2022
It's possible to do, but will be a little slow due to the nature of tar files. They're a big linked list of files, so you need to do a bunch of reads/seeks from the start to the end to enumerate all files.
I'd ask that we try to get bpo-24132 solved first. That would let us write:
We'd fill in a smallish number of abstract methods to get a full
Path-compatible class withread_text(),is_symlink()etc methods.jaraco commentedon Jan 1, 2022
I'd recommend not to block on bpo-24132. It's not obvious to me that subclassing would be valuable. It depends on how it's implemented, but in my experience, zipfile.Path doesn't and cannot implement the full interface of pathlib.Path. Instead zipfile.Path attempts to implement a protocol. At the time, the protocol was undefined, but now there exists importlib.resources.abc.Traversable (https://docs.python.org/3/library/importlib.html#importlib.abc.Traversable), the interface needed by importlib.resources. I'd honestly just create a standalone class, see if it can implement Traversable, and only then consider if it should implement a more complicated interface (such as something with symlink support or perhaps even later subclassing from pathlib.Path).
barneygale commentedon Jan 2, 2022
If you're only aiming for Traversable compatibility, sure.
The original bug description asks for something that's pathlib-compatible and similar to zipfile.Path, which goes beyond the Traversable interface in attempting to emulate pathlib.Path.
The pathlib.Path interface is a good one - I see no reason it can't apply to zip and tar archives in full. Methods of Path objects already raise NotImplementedError if operations aren't supported (e.g. creating symlinks)
Some prototyping from a couple years back, including a tar path implementation: https://github.com/barneygale/pathlab/tree/master/pathlab
encukou commentedon May 3, 2023
Anyone implementing this should be aware that tar members don't have unique names.
A path should generally refer to the last member of the given name (which would overwrite previous ones when extracting -- unless the previous members do shenanigans with symlinks/directories, but API for accessing individual members isn't expected to take that into account).
TarFile API like
getmemberwill take care of most of this, but not all: for example, a “listdir” operation should probably de-duplicate its result.30 remaining items
GH-89812: Improve test for `pathlib.Path.stat()` (GH-106064)
pythonGH-89812: Simplify creation of symlinks in pathlib tests. (pyth…
Merge branch 'main' into pythongh-89812-omgtarpath
barneygale commentedon Jul 3, 2023
PR available that adds
tarfile.TarPath! #106337barneygale commentedon Jul 9, 2023
What should
tarfile.TarPathdo with symlinks in archives? In the PR above they're followed in the same circumstances as inpathlib.PosixPath, but that might not be desirable/safe default. We could add an initialiser argument that disabled following symlinks entirely - either omit them or represent them as regular files, somehow?If the answer isn't clear then I'd like to incubate this in a PyPI project for a while first
@encukou I saw you were working on tarfile recently. Pinging in case this interests you!
encukou commentedon Jul 10, 2023
IMO, symlinks to files that are in the archive should continue to work.
Perhaps relative symlinks to files within the archive root should work too, though these become dangerous if there are any directory symlinks pointing outside.
In general, if an archive contains directory symlinks, you can only get the final destination of a file by extracting the archive (or by doing a very elaborate dry run, for which
tarfileisn't suited).It's very much not clear-cut. A tarball is, essentially, a collection of procedural instructions that allow merging contents with an existing filesystem in ”interesting” ways, while
TarPathwould like to work with a final tree structure.IMO, to answer questions like this, we first need a good conceptual model of what TarPath represents.
A PyPI project that handles the simple cases sounds like a good start.
barneygale commentedon Jul 12, 2023
Thank you, @encukou! I'll move the
TarPathimplementation in #106337 into an experimental PyPI package then, leaving just the (private)pathlib._VirtualPathin that PR.Merge branch 'main' into pythongh-89812-omgtarpath
encukou commentedon Jul 13, 2023
FWIW, I won't make one any time soon -- I'm taking a break from
tarfileafter a few months of dealing with it.But if you want to bounce ideas off me, I'm here.
Merge branch 'main' into pythongh-89812-omgtarpath
Merge branch 'main' into pythongh-89812-omgtarpath
Merge branch 'main' into pythongh-89812-omgtarpath
Merge branch 'main' into pythongh-89812-omgtarpath
GH-89812: Add `pathlib._PathBase` (#106337)
pythonGH-89812: Add `pathlib._PathBase` (python#106337)