安装Lance Python SDK

pip install https://proton-pkgs.tos-cn-beijing.volces.com/lance/pylance-0.18.2-cp39-abi3-manylinux_2_28_x86_64.whl

读取案例

import lance
ds = lance.dataset(
    "s3://{PATH}",
    storage_options={
        "access_key_id": "<用户实际的AK>",
        "secret_access_key": "<用户实际的SK>",
        "aws_endpoint": "https://bucket1.tos-s3-cn-shanghai.ivolces.com",
        "virtual_hosted_style_request": "true"
    }
)
print(ds.to_table().to_pandas())

参数说明

• PATH: 对象存储的路径, 包含桶名, 例如s3://bucket1/test/lance.db。
• AK/SK: 访问秘钥。
• aws_endpoint: TOS的Endpoint的, 必须使用https协议, Endpoint路径中必须包含桶名, 为VirtualHost模式。
• virtual_hosted_style_request: 必须为True。

写入案例

import lance
import pyarrow as pa

table = pa.Table.from_pylist([{"name": "Alice", "age": 20},
                              {"name": "Bob", "age": 30}])
schema = pa.schema([
        pa.field("name", pa.string()),
        pa.field("age", pa.int64()),
    ])

storage_options={
        "access_key_id": "<用户实际的AK>",
        "secret_access_key": "<用户实际的SK>",
        "aws_endpoint": "https://bucket1.tos-s3-cn-shanghai.ivolces.com",
        "virtual_hosted_style_request": "true"
    }
lance.write_dataset(table, "s3://{PATH}", schema, storage_options=storage_options)