zarr.testing.buffer

Classes

NDBufferUsingTestNDArrayLike	Example of a custom NDBuffer that handles MyNDArrayLike
StoreExpectingTestBuffer	Example of a custom Store that expect MyBuffer for all its non-metadata
TestBuffer	Example of a custom Buffer that handles ArrayLike

Module Contents

class zarr.testing.buffer.NDBufferUsingTestNDArrayLike(array: zarr.core.buffer.core.NDArrayLike)

Bases: zarr.core.buffer.cpu.NDBuffer

Example of a custom NDBuffer that handles MyNDArrayLike

all_equal(other: Any, equal_nan: bool = True) → bool

Compare to other using np.array_equal.

as_ndarray_like() → NDArrayLike

Returns the underlying array (host or device memory) of this buffer

This will never copy data.

Returns:

The underlying array such as a NumPy or CuPy array.

as_numpy_array() → numpy.typing.NDArray[Any]

Returns the buffer as a NumPy array (host memory).

Returns:

NumPy array of this buffer (might be a data copy)

Warning

Might have to copy data, consider using .as_ndarray_like() instead.

as_scalar() → ScalarType

Returns the buffer as a scalar value

astype(

dtype: numpy.typing.DTypeLike,

order: Literal['K', 'A', 'C', 'F'] = 'K',

) → Self

copy() → Self

classmethod create(

*,

shape: collections.abc.Iterable[int],

dtype: numpy.typing.DTypeLike,

order: Literal['C', 'F'] = 'C',

fill_value: Any | None = None,

) → Self

Overwrite NDBuffer.create to create an TestNDArrayLike instance

classmethod empty(

shape: zarr.core.common.ChunkCoords,

dtype: numpy.typing.DTypeLike,

order: Literal['C', 'F'] = 'C',

) → Self

Create an empty buffer with the given shape, dtype, and order.

This method can be faster than NDBuffer.create because it doesn’t have to initialize the memory used by the underlying ndarray-like object.

Parameters:

shape

The shape of the buffer and its underlying ndarray-like object

dtype

The datatype of the buffer and its underlying ndarray-like object

order

Whether to store multi-dimensional data in row-major (C-style) or column-major (Fortran-style) order in memory.

Returns:

buffer

New buffer representing a new ndarray_like object with empty data.

See also

NDBuffer.create

Create a new buffer with some initial fill value.

fill(value: Any) → None

classmethod from_ndarray_like(ndarray_like: NDArrayLike) → Self

Create a new buffer of a ndarray-like object

Parameters:

ndarray_like

ndarray-like object

Returns:

New buffer representing ndarray_like

classmethod from_numpy_array(array_like: numpy.typing.ArrayLike) → Self

Create a new buffer of Numpy array-like object

Parameters:

array_like

Object that can be coerced into a Numpy array

Returns:

New buffer representing array_like

reshape(newshape: zarr.core.common.ChunkCoords | Literal[-1]) → Self

squeeze(axis: tuple[int, Ellipsis]) → Self

transpose(

axes: SupportsIndex | collections.abc.Sequence[SupportsIndex] | None,

) → Self

property byteorder: zarr.codecs.bytes.Endian

property dtype: numpy.dtype[Any]

property shape: tuple[int, Ellipsis]

class zarr.testing.buffer.StoreExpectingTestBuffer(

store_dict: collections.abc.MutableMapping[str, zarr.core.buffer.Buffer] | None = None,

*,

read_only: bool = False,

)

Bases: zarr.storage.MemoryStore

Example of a custom Store that expect MyBuffer for all its non-metadata

We assume that keys containing “json” is metadata

async clear() → None

Clear the store.

Remove all keys and values from the store.

close() → None

Close the store.

async delete(key: str) → None

Remove a key from the store

Parameters:

keystr

async delete_dir(prefix: str) → None

Remove all keys and prefixes in the store that begin with a given prefix.

async exists(key: str) → bool

Check if a key exists in the store.

Parameters:

keystr

Returns:

bool

async get(

key: str,

prototype: zarr.core.buffer.BufferPrototype,

byte_range: tuple[int, int | None] | None = None,

) → zarr.core.buffer.Buffer | None

Retrieve the value associated with a given key.

Parameters:

keystr

prototypeBufferPrototype

The prototype of the output buffer. Stores may support a default buffer prototype.

byte_rangeByteRequest, optional

ByteRequest may be one of the following. If not provided, all data associated with the key is retrieved. - RangeByteRequest(int, int): Request a specific range of bytes in the form (start, end). The end is exclusive. If the given range is zero-length or starts after the end of the object, an error will be returned. Additionally, if the range ends after the end of the object, the entire remainder of the object will be returned. Otherwise, the exact requested range will be returned. - OffsetByteRequest(int): Request all bytes starting from a given byte offset. This is equivalent to bytes={int}- as an HTTP header. - SuffixByteRequest(int): Request the last int bytes. Note that here, int is the size of the request, not the byte offset. This is equivalent to bytes=-{int} as an HTTP header.

Returns:

Buffer

async get_partial_values(

prototype: zarr.core.buffer.BufferPrototype,

key_ranges: collections.abc.Iterable[tuple[str, zarr.abc.store.ByteRequest | None]],

) → list[zarr.core.buffer.Buffer | None]

Retrieve possibly partial values from given key_ranges.

Parameters:

prototypeBufferPrototype

The prototype of the output buffer. Stores may support a default buffer prototype.

key_rangesIterable[tuple[str, tuple[int | None, int | None]]]

Ordered set of key, range pairs, a key may occur multiple times with different ranges

Returns:

list of values, in the order of the key_ranges, may contain null/none for missing keys

async getsize(key: str) → int

Return the size, in bytes, of a value in a Store.

Parameters:

keystr

Returns:

nbytesint

The size of the value (in bytes).

Raises:

FileNotFoundError

When the given key does not exist in the store.

async getsize_prefix(prefix: str) → int

Return the size, in bytes, of all values under a prefix.

Parameters:

prefixstr

The prefix of the directory to measure.

Returns:

nbytesint

The sum of the sizes of the values in the directory (in bytes).

See also

zarr.Array.nbytes_stored

Store.getsize

Notes

getsize_prefix is just provided as a potentially faster alternative to listing all the keys under a prefix calling Store.getsize() on each.

In general, prefix should be the path of an Array or Group in the Store. Implementations may differ on the behavior when some other prefix is provided.

async is_empty(prefix: str) → bool

Check if the directory is empty.

Parameters:

prefixstr

Prefix of keys to check.

Returns:

bool

True if the store is empty, False otherwise.

async list() → collections.abc.AsyncIterator[str]

Retrieve all keys in the store.

Returns:

AsyncIterator[str]

async list_dir(prefix: str) → collections.abc.AsyncIterator[str]

Retrieve all keys and prefixes with a given prefix and which do not contain the character “/” after the given prefix.

Parameters:

prefixstr

Returns:

AsyncIterator[str]

async list_prefix(prefix: str) → collections.abc.AsyncIterator[str]

Retrieve all keys in the store that begin with a given prefix. Keys are returned relative to the root of the store.

Parameters:

prefixstr

Returns:

AsyncIterator[str]

classmethod open(*args: Any, **kwargs: Any) → Self

Async:

Create and open the store.

Parameters:

*argsAny

Positional arguments to pass to the store constructor.

**kwargsAny

Keyword arguments to pass to the store constructor.

Returns:

Store

The opened store instance.

async set(

key: str,

value: zarr.core.buffer.Buffer,

byte_range: tuple[int, int] | None = None,

) → None

Store a (key, value) pair.

Parameters:

keystr

valueBuffer

async set_if_not_exists(key: str, value: zarr.core.buffer.Buffer) → None

Store a key to value if the key is not already present.

Parameters:

keystr

valueBuffer

abstract set_partial_values(

key_start_values: collections.abc.Iterable[tuple[str, int, bytes | bytearray | memoryview[int]]],

) → None

Async:

Store values at a given key, starting at byte range_start.

Parameters:

key_start_valueslist[tuple[str, int, BytesLike]]

set of key, range_start, values triples, a key may occur multiple times with different range_starts, range_starts (considering the length of the respective values) must not specify overlapping ranges for the same key

with_read_only(read_only: bool = False) → MemoryStore

Return a new store with a new read_only setting.

The new store points to the same location with the specified new read_only state. The returned Store is not automatically opened, and this store is not automatically closed.

Parameters:

read_only

If True, the store will be created in read-only mode. Defaults to False.

Returns:

A new store of the same type with the new read only attribute.

property read_only: bool

Is the store read-only?

property supports_consolidated_metadata: bool

Does the store support consolidated metadata?.

If it doesn’t an error will be raised on requests to consolidate the metadata. Returning False can be useful for stores which implement their own consolidation mechanism outside of the zarr-python implementation.

supports_deletes: bool = True

Does the store support deletes?

supports_listing: bool = True

Does the store support listing?

supports_partial_writes: bool = True

Does the store support partial writes?

supports_writes: bool = True

Does the store support writes?

class zarr.testing.buffer.TestBuffer(array_like: zarr.core.buffer.core.ArrayLike)

Bases: zarr.core.buffer.cpu.Buffer

Example of a custom Buffer that handles ArrayLike

as_array_like() → ArrayLike

Returns the underlying array (host or device memory) of this buffer

This will never copy data.

Returns:

The underlying 1d array such as a NumPy or CuPy array.

as_buffer_like() → zarr.core.common.BytesLike

Returns the buffer as an object that implements the Python buffer protocol.

Returns:

An object that implements the Python buffer protocol

Notes

Might have to copy data, since the implementation uses .as_numpy_array().

as_numpy_array() → numpy.typing.NDArray[Any]

Returns the buffer as a NumPy array (host memory).

Returns:

NumPy array of this buffer (might be a data copy)

Notes

Might have to copy data, consider using .as_array_like() instead.

classmethod create_zero_length() → Self

Create an empty buffer with length zero

Returns:

New empty 0-length buffer

classmethod from_array_like(array_like: ArrayLike) → Self

Create a new buffer of an array-like object

Parameters:

array_like

array-like object that must be 1-dim, contiguous, and byte dtype.

Returns:

New buffer representing array_like

classmethod from_buffer(buffer: zarr.core.buffer.core.Buffer) → Self

Create a new buffer of an existing Buffer

This is useful if you want to ensure that an existing buffer is of the correct subclass of Buffer. E.g., MemoryStore uses this to return a buffer instance of the subclass specified by its BufferPrototype argument.

Typically, this only copies data if the data has to be moved between memory types, such as from host to device memory.

Parameters:

buffer

buffer object.

Returns:

A new buffer representing the content of the input buffer

Notes

Subclasses of Buffer must override this method to implement more optimal conversions that avoid copies where possible

classmethod from_bytes(bytes_like: zarr.core.common.BytesLike) → Self

Create a new buffer of a bytes-like object (host memory)

Parameters:

bytes_like

bytes-like object

Returns:

New buffer representing bytes_like

to_bytes() → bytes

Returns the buffer as bytes (host memory).

Returns:

bytes of this buffer (data copy)

Warning

Will always copy data, only use this method for small buffers such as metadata buffers. If possible, use .as_numpy_array() or .as_array_like() instead.