File: //opt/gsutil/third_party/apitools/apitools/base/py/__pycache__/transfer.cpython-39.pyc
a
�z�g�� � @ sF d Z ddlmZ ddlmZ ddlm mZ ddl
m m Z ddl
Z
ddlZddlZddlZddlZddlZddlmZ ddlmZ ddlmZ ddlmZ ddlmZ dd lmZ dd
lmZ g d �Z d Z d
Z dZ dd� Z dd� Z!dd� Z"dd� Z#G dd� de$�Z%G dd� de%�Z&ej'�r2G dd � d ej(�Z)G d d � d e%�Z*dS ) z)Upload and download support for apitools.� )�print_functionN)� http_client)�buffered_stream)� compression)�
exceptions)� http_wrapper)� stream_slice)�util)�Download�Upload�RESUMABLE_UPLOAD�
SIMPLE_UPLOAD�DownloadProgressPrinter�DownloadCompletePrinter�UploadProgressPrinter�UploadCompletePrinteri P �simple� resumablec C s0 d| j v r td| j d � ntd| j � dS )z*Print download progress based on response.�
content-rangez Received %szReceived %d bytesN)�info�print�length)�response�unused_download� r �=/opt/gsutil/third_party/apitools/apitools/base/py/transfer.pyr 7 s
r c C s t d� dS )z-Print information about a completed download.zDownload completeN�r )�unused_responser r r r r ? s r c C s t d| jd � dS )z(Print upload progress based on response.zSent %s�rangeN)r r )r �
unused_uploadr r r r D s r c C s t d� dS )z+Print information about a completed upload.zUpload completeNr )r r r r r r I s r c @ s� e Zd ZdZd&dd�Zdd � Zed
d � �Zed d
� �Zedd� �Z e j
dd� �Z edd� �Z e j
dd� �Z edd� �Z edd� �Z
dd� Zedd� �Zed d � �Zd d � Zd d!� Zd"d#� Zd$d%� ZdS )'� _Transferz-Generic bits common to Uploads and Downloads.FNT� c C sF d | _ || _|| _|| _d | _d| _|| _tj| _ || _
|p>d| _ d S )Nr! i ) �_Transfer__bytes_http�_Transfer__close_stream�_Transfer__http�_Transfer__stream�_Transfer__url�_Transfer__num_retries� num_retriesr �)HandleExceptionsAndRebuildHttpConnections�
retry_func�
auto_transfer� chunksize)�self�stream� close_streamr, r+ �httpr( r r r �__init__R s �z_Transfer.__init__c C s t | �S �N)�str�r- r r r �__repr__c s z_Transfer.__repr__c C s | j S r2 )r# r4 r r r r/ f s z_Transfer.close_streamc C s | j S r2 )r$ r4 r r r r0 j s z_Transfer.httpc C s | j p
| jS r2 )r" r0 r4 r r r �
bytes_httpn s z_Transfer.bytes_httpc C s
|| _ d S r2 )r" �r- �valuer r r r6 r s c C s | j S r2 )r' r4 r r r r( v s z_Transfer.num_retriesc C s* t �|tj� |dk r t�d��|| _d S )Nr z*Cannot have negative value for num_retries)r � Typecheck�six�
integer_typesr �InvalidDataErrorr' r7 r r r r( z s �c C s | j S r2 )r% r4 r r r r. � s z_Transfer.streamc C s | j S r2 )r& r4 r r r �url� s z
_Transfer.urlc C s* | � � | jdu r |p t�� | _|| _dS )a� Initialize this download by setting self.http and self.url.
We want the user to be able to override self.http by having set
the value in the constructor; in that case, we ignore the provided
http.
Args:
http: An httplib2.Http instance or None.
url: The url for this transfer.
Returns:
None. Initializes self.
N)�EnsureUninitializedr0 r �GetHttpr$ r& )r- r0 r= r r r � _Initialize� s
z_Transfer._Initializec C s | j d uo| jd uS r2 )r= r0 r4 r r r � initialized� s z_Transfer.initializedc C s
t | �jS r2 )�type�__name__r4 r r r �
_type_name� s z_Transfer._type_namec C s | j st�d| j ��d S )NzCannot use uninitialized %s�rA r �TransferInvalidErrorrD r4 r r r �EnsureInitialized� s �z_Transfer.EnsureInitializedc C s | j rt�d| j ��d S )NzCannot re-initialize %srE r4 r r r r> � s �z _Transfer.EnsureUninitializedc C s | j r| j�� d S r2 )r# r% �closer4 r r r �__del__� s z_Transfer.__del__c C s" |d ur t j||| fd��� d S )N)�target�args)� threading�Thread�start)r- �callbackr r r r �_ExecuteCallback� s z_Transfer._ExecuteCallback)FNTNr! )rC �
__module__� __qualname__�__doc__r1 r5 �propertyr/ r0 r6 �setterr( r. r= r@ rA rD rG r> rI rP r r r r r N s< �
r c s e Zd ZdZeejejejej f�Z
ed�Z d.� fdd� Z e
dd� �Ze
dd � �Zed/d d
��Zed0dd��Zed1dd��Ze
dd� �Ze
dd� �Zdd� Zdd� Zdd� Zd2d d �Zd3d d �Zd4d d!�Zd5d"d#�Z d6d$d%�Z d&d'� Z d7d(d)�Z d8d*d+�Z d9d,d-�Z!� Z"S ):r
zqData for a single download.
Public attributes:
chunksize: default chunksize to use for transfers.
�r+ �progress�
total_sizer= Nc sL |� dd �}tt| �j|fi |�� d | _d| _|| _d | _|| _|| _ d S )NrX r )
�pop�superr
r1 �_Download__initial_response�_Download__progress�_Download__total_size�_Download__encoding�progress_callback�finish_callback)r- r. r_ r` �kwdsrX �� __class__r r r1 � s zDownload.__init__c C s | j S r2 )r\ r4 r r r rW � s zDownload.progressc C s | j S r2 )r^ r4 r r r �encoding� s zDownload.encodingFTc K sF t j�|�}t j�|�r*|s*t�d| ��| t|d�fd|d�|��S )z-Create a new download object from a filename.z*File %s exists and overwrite not specified�wbT)r/ r+ )�os�path�
expanduser�existsr �InvalidUserInputError�open)�cls�filenameZ overwriter+ ra rg r r r �FromFile� s ���zDownload.FromFilec K s | |f||d�|��S )z+Create a new Download object from a stream.)r+ rX r )rl r. r+ rX ra r r r �
FromStream� s �zDownload.FromStreamc K s� t �|�}| jt|�� � }|r4t�dd�|� ��| j|fi |��} |durV|| _ n
|d | _ |durx|�
|d �}
n|d }
t | d|d � t | d|d � | � ||
� | S )
z?Create a new Download object from a stream and serialized data.�,Invalid serialization data, missing keys: %s�, Nr+ r= r\ rW r] rX )
�json�loads� _REQUIRED_SERIALIZATION_KEYS�set�keysr r< �joinro r+ �FinalizeTransferUrl�setattrr@ ) rl r. � json_datar0 r+ �clientra r � missing_keys�downloadr= r r r �FromData� s*
��
�zDownload.FromDatac C s | � � | j| j| j| jd�S )NrV )rG r+ rW rX r= r4 r r r �serialization_data s �zDownload.serialization_datac C s | j S r2 )r] r4 r r r rX s zDownload.total_sizec C s | j s
dS d| j| j| jf S )NzDownload (uninitialized)z1Download with %d/%s bytes transferred from url %s�rA rW rX r= r4 r r r �__str__ s
�zDownload.__str__c C s$ d|j d< d| jd f |jd< d S )N�media�altz
bytes=0-%d� �Range)� query_paramsr, �headers)r- � http_request� url_builderr r r �ConfigureRequest s
zDownload.ConfigureRequestc C sB d|v r.|d � d�\}}}|dkr.t|�| _| jdu r>d| _dS )z;Sets the total size based off info if possible otherwise 0.r �/�*Nr )�
rpartition�intr] rX )r- r �_�totalr r r Z
__SetTotal# s
zDownload.__SetTotalc C s� | � � |du r"|du r"t�d��|p*|j}|durB|�|j�|_|j}| jr�| �d�}| �|d|� t �
| j pr||�}|j | j
vr�tj�|��|| _| �|j� |j�d|j�}|dur�|�|�}| �||� | jr�| �� dS )a� Initialize this download by making a request.
Args:
http_request: The HttpRequest to use to initialize this download.
http: The httplib2.Http instance for this request.
client: If provided, let this client process the final URL before
sending any additional requests. If client is provided and
http is not, client.http will be used instead.
N� Must provide client or http.r zcontent-location)r> r � UserErrorr0 rx r= r+ �_Download__ComputeEndByte�_Download__SetRangeHeaderr � MakeRequestr6 � status_code�_ACCEPTABLE_STATUSES� HttpError� FromResponser[ �_Download__SetTotalr �get� request_urlr@ �StreamInChunks)r- r� r0 r{ r= �end_byter r r r �InitializeDownload0 s.
�
zDownload.InitializeDownloadc C s� |dur||dk r(t �dd||f ��n"|| jkrJt �dd|| jf ��t|| jd �}||k rtt �d||f ��||fS |dk r�td|| j �}|| jd fS dS ) z4Normalizes start and end values based on total size.Nr z0Cannot have end index with negative start index z[start=%d, end=%d]z0Cannot have start index greater than total size z[start=%d, total_size=%d]r� z(Range requested with end[%s] < start[%s])r rF rX �min�max)r- rN �endr r r Z__NormalizeStartEndS s,
��
��
�z Download.__NormalizeStartEndc C sN |dk rd| |j d< n2|d u s(||k r8d| |j d< nd||f |j d< d S )Nr zbytes=%dr z bytes=%d-z bytes=%d-%d)r� )r- �requestrN r� r r r Z__SetRangeHeaderh s
zDownload.__SetRangeHeaderc C sl |}|dk r| j s|S |r@|| j d }|dur<t||�}n|}| j rh| j d }|durdt||�}n|}|S )a� Compute the last byte to fetch for this request.
This is all based on the HTTP spec for Range and
Content-Range.
Note that this is potentially confusing in several ways:
* the value for the last byte is 0-based, eg "fetch 10 bytes
from the beginning" would return 9 here.
* if we have no information about size, and don't want to
use the chunksize, we'll return None.
See the tests for more examples.
Args:
start: byte to start at.
end: (int or None, default: None) Suggested last byte.
use_chunks: (bool, default: True) If False, ignore self.chunksize.
Returns:
Last byte to use in a Range header, or None.
r r� N)rX r, r� )r- rN r� �
use_chunksr� � alternater r r Z__ComputeEndBytep s
zDownload.__ComputeEndBytec C sR | � � tj| jd�}| j|||d� |dur:|j�|� tj| j|| j | j
d�S )z/Retrieve a chunk, and return the full response.)r= )r� N)r* �retries) rG r �Requestr= r� r� �updater� r6 r* r( )r- rN r� �additional_headersr� r r r Z
__GetChunk� s
�zDownload.__GetChunkc C s� |j | jvr8|j tjtjfv r,tj�|��n t�|j ��|j tj
tj fv r�z| j �
t�|j �� W n& ty� | j �
t�|j �� Y n0 | j|j7 _|jr�d|jv r�|jd | _n|j tjkr�| j �
d� |S )z?Process response (by updating self and writing to self.stream).zcontent-encoding� )r� r� r � FORBIDDEN� NOT_FOUNDr r� r� �TransferRetryError�content�OK�PARTIAL_CONTENTr. �writer: �
ensure_binary� TypeError� ensure_textr\ r r r^ �
NO_CONTENT�r- r r r r Z__ProcessResponse� s& � � zDownload.__ProcessResponsec C s� | � � d}| jdur,| �||�\}}d}n|}|}|rH|du sH||kr�| j|||d�}| j|||d�}|s�| �|j� | �||�\}}d}| �|�}||j7 }|jdkr4|j t
j kr�dS t �
d��q4dS )a� Retrieve a given byte range from this download, inclusive.
Range must be of one of these three forms:
* 0 <= start, end = None: Fetch from start to the end of the file.
* 0 <= start <= end: Fetch the bytes from start to end.
* start < 0, end = None: Fetch the last -start bytes of the file.
(These variations correspond to those described in the HTTP 1.1
protocol for range headers in RFC 2616, sec. 14.35.1.)
Args:
start: (int) Where to start fetching bytes. (See above.)
end: (int, optional) Where to stop fetching bytes. (See above.)
additional_headers: (bool, optional) Any additional headers to
pass with the request.
use_chunks: (bool, default: True) If False, ignore self.chunksize
and fetch this range in a single request.
Returns:
None. Streams bytes into self.stream.
FNT)r� r� �r� r z5Zero bytes unexpectedly returned in download response)rG rX � _Download__NormalizeStartEndr� �_Download__GetChunkr� r �_Download__ProcessResponser r� r r� r r� ) r- rN r� r� r� Zprogress_end_normalizedrW r� r r r r �GetRange� s8
���
�zDownload.GetRangec C s | j |||dd� dS )z%Stream the entire download in chunks.T�rO r` r� r� N)� StreamMedia�r- rO r` r� r r r r� � s �zDownload.StreamInChunksc C s� |p| j }|p| j}| �� | jdur4| j}d| _n"| j| j|d�}| j| j||d�}| jdu rl| �|j � | �
|�}| � ||� |j t
jks�| j| jkr q�q | � ||� dS )a4 Stream the entire download.
Args:
callback: (default: None) Callback to call as each chunk is
completed.
finish_callback: (default: None) Callback to call when the
download is complete.
additional_headers: (default: None) Additional headers to
include in fetching bytes.
use_chunks: (bool, default: True) If False, ignore self.chunksize
and stream this download in a single request.
Returns:
None. Streams bytes into self.stream.
N)r� r� )r_ r` rG r[ r� rW r� rX r� r r� rP r� r r� )r- rO r` r� r� r r� r r r r� � s,
��
�zDownload.StreamMedia)NN)FT)TN)NNN)NN)N)N)NT)N)NNT)NNN)NNNT)#rC rQ rR rS ru r r� r� r� � REQUESTED_RANGE_NOT_SATISFIABLEr� rt r1 rT rW rd � classmethodrn ro r~ r rX r� r� r� r� r� r� r� r� r� r� r� r� �
__classcell__r r rb r r
� sV �
�
#
+
�
6 �
�r
c @ s$ e Zd ZdZdd� Zdd� ZeZdS )�MultipartBytesGeneratorao Generates a bytes Message object tree for multipart messages
This is a BytesGenerator that has been modified to not attempt line
termination character modification in the bytes payload. Known to
work with the compat32 policy only. It may work on others, but not
tested. The outfp object must accept bytes in its write method.
c C s |j d u rd S | �|j � d S r2 )�_payloadr� )r- �msgr r r � _handle_text1 s
z$MultipartBytesGenerator._handle_textc C s |� dd�S )N�ascii�surrogateescape)�encode)r- �sr r r �_encode8 s z MultipartBytesGenerator._encodeN)rC rQ rR rS r� r� �
_writeBodyr r r r r� ) s r� c sV e Zd ZdZed�Zd@� fdd� Zedd � �Ze dAd
d ��Z
e dBd d
��Z e dCdd��Z edd� �Z
edd� �Zedd� �Zdd� Zedd� �Zejdd� �Zedd � �Zejd d � �Zd d � Zd d!� Zd"d#� Zd$d%� Zd&d'� Zd(d)� Zd*d+� ZdDd,d-�Zd.d/� Z dEd0d1�Z d2d3� Z dFd4d5�Z dGd6d7�Z dHd8d9�Z!d:d;� Z"dId<d=�Z#dJd>d?�Z$� Z%S )Kr a� Data for a single Upload.
Fields:
stream: The stream to upload.
mime_type: MIME type of the upload.
total_size: (optional) Total upload size for the stream.
close_stream: (default: False) Whether or not we should close the
stream when finished with the upload.
auto_transfer: (default: True) If True, stream all bytes as soon as
the upload is created.
�r+ � mime_typerX r= NFTc sh t t| �j|f||||d�| �� d| _d | _|| _d| _d | _d | _d | _ |
| _
|| _ | | _ || _
d S )N)r/ r, r+ r0 Fr )rZ r r1 �_Upload__complete�_Upload__final_response�_Upload__mime_type�_Upload__progress�!_Upload__server_chunk_granularity�_Upload__strategy�_Upload__total_size�_Upload__gzip_encodedr_ r` rX ) r- r. r� rX r0 r/ r, r+ r_ r` � gzip_encodedra rb r r r1 O s&
���zUpload.__init__c C s | j S r2 )r� r4 r r r rW c s zUpload.progressc K s| t j�|�}t j�|�s&t�d| ��|sNt�|�\}}|du rNt�d| ��t � |�j
}| t |d�|f|d||d�|��S )z+Create a new Upload object from a filename.zCould not find file %sNz Could not guess mime type for %s�rbT�rX r/ r+ r� ) rf rg rh ri r �
NotFoundError� mimetypes�
guess_typerj �stat�st_sizerk ) rl rm r� r+ r� ra rg r� �sizer r r rn g s � ��zUpload.FromFilec K s. |du rt �d��| ||f|d||d�|��S )z)Create a new Upload object from a stream.Nz!No mime_type specified for streamFr� )r rj )rl r. r� rX r+ r� ra r r r ro x s �
��zUpload.FromStreamc K s� t �|�}| jt|�� � } | r4t�dd�| � ��d|v rFt�d��| j ||d f|�
d�|d�|��}
t |t j
�r�|�� s�t�d��|dur�||
_n
|d |
_|dur�|�|d
�} n|d
} t|
_|
�|| � |
�� |
�� |
jr�|
�� |
S ) zACreate a new Upload of stream from serialized json_data and http.rp rq rX z/Cannot override total_size on serialized Uploadr� )rX r� z6Cannot restart resumable upload on non-seekable streamNr+ r= )rr rs rt ru rv r r< rw rj ro r� �
isinstance�io�IOBase�seekabler+ rx r �strategyr@ �RefreshResumableUploadStaterG r� ) rl r. rz r0 r+ r� r{ ra r r| �uploadr= r r r r~ � sJ
������
�zUpload.FromDatac C s2 | � � | jtkr t�d��| j| j| j| jd�S )Nz2Serialization only supported for resumable uploadsr� ) rG r� r r r< r+ r� rX r= r4 r r r r � s
��zUpload.serialization_datac C s | j S r2 )r� r4 r r r �complete� s zUpload.completec C s | j S r2 )r� r4 r r r r� � s zUpload.mime_typec C s" | j s
dS d| j| jpd| jf S )NzUpload (uninitialized)z.Upload with %d/%s bytes transferred for url %sz???r� r4 r r r r� � s
�zUpload.__str__c C s | j S r2 )r� r4 r r r r� � s zUpload.strategyc C s$ |t tfvrt�d| ��|| _d S )NzOInvalid value "%s" for upload strategy, must be one of "simple" or "resumable".)r
r r r� r� r7 r r r r� � s �c C s | j S r2 )r� r4 r r r rX � s zUpload.total_sizec C s | � � || _d S r2 )r> r� r7 r r r rX � s c C s^ |j du rt| _| jdur dS t}| jdur:| jtkr:t}|jrJ|jsJt}|jsTt}|| _dS )a Determine and set the default upload strategy for this upload.
We generally prefer simple or multipart, unless we're forced to
use resumable. This happens when any of (1) the upload is too
large, (2) the simple endpoint doesn't support multipart requests
and we have metadata, or (3) there is no simple upload endpoint.
Args:
upload_config: Configuration for the upload endpoint.
http_request: The associated http request.
Returns:
None.
N) �resumable_pathr
r� rX �_RESUMABLE_UPLOAD_THRESHOLDr �body�simple_multipart� simple_path)r- �
upload_configr� r� r r r Z__SetDefaultUploadStrategy� s
� z!Upload.__SetDefaultUploadStrategyc C s� | j r.|jr.| j |jkr.t�d| j |jf ��t�|j| j�sTt�d| j|jf ��| �||� | j t
kr�|j |_ |j
r�d|jd< | �|� nd|jd< | �|� | jr�d|jd< t�t�|j
��d �� |_
n |j|_ d |jd< | �|� d
S ) z.Configure the request and url for this upload.z*Upload too big: %s larger than max size %sz7MIME type %s does not match any accepted MIME ranges %s� multipartZ
uploadTyper� �gzip�Content-Encodingr r N)rX �max_sizer rj r �AcceptableMimeType�acceptr� �!_Upload__SetDefaultUploadStrategyr� r
r� �
relative_pathr� r� �"_Upload__ConfigureMultipartRequest� _Upload__ConfigureMediaRequestr� r� r �CompressStreamr: �BytesIO�readr� �"_Upload__ConfigureResumableRequest)r- r� r� r� r r r r� � sB
�����
���
zUpload.ConfigureRequestc C s" | j |jd< | j�� |_d|_dS )z;Configure http_request as a simple request for this upload.� content-type� <media body>N)r� r� r. r� r� �
loggable_body�r- r� r r r Z__ConfigureMediaRequest" s z Upload.__ConfigureMediaRequestc C s& t �d�}t|ddd� � tj|jd �d�� }|�|j� |� |� tj| j
�d�� }d|d< |�| j � � � |� |� t
�� }t
jr�t}ntj}||d d
�}|j|d d � |�� |_|�� }d | |jd< t|t
j�r�|�d
�}|j�|�}|d �d�\} }
}
d�| dg�|d< |�|�|_dS )z>Configure http_request as a multipart request for this upload.�related�_write_headersc S s d S r2 r r4 r r r �<lambda>- � z4Upload.__ConfigureMultipartRequest.<locals>.<lambda>r� r� �binaryzContent-Transfer-EncodingF)� mangle_from_)�unixfromz multipart/related; boundary=%rr� ���s
s <media body>
--N) �mime_multipart�
MIMEMultipartry �mime_nonmultipart�MIMENonMultipartr� �split� set_payloadr� �attachr� r. r� r: r� �PY3r� �email_generator� Generator�flatten�getvalue� get_boundaryr� � text_typer� � partitionrw r� ) r- r� Zmsg_rootr� �fpZgenerator_class�gZmultipart_boundaryZbody_componentsr� r� r r r Z__ConfigureMultipartRequest( s6
�
�
z"Upload.__ConfigureMultipartRequestc C s* | j |jd< | jd ur&t| j�|jd< d S )NzX-Upload-Content-TypezX-Upload-Content-Length)r� r� rX r3 r� r r r Z__ConfigureResumableRequestR s
��z"Upload.__ConfigureResumableRequestc C s� | j tkrdS | �� tj| jdddid�}tj| j|d| jd�}| � |�}|j
t j t j
fv r�d| _| j| _| j�| j� || _nH|j
tjkr�|du r�d| _n| �|�d | _| j�| j� n tj�|��dS )
z�Talk to the server and refresh the state of this resumable upload.
Returns:
Response if the upload is complete.
N�PUT�
Content-Rangez bytes */*)r= � http_methodr� r )� redirectionsr� Tr� )r� r rG r r� r= r� r0 r( �_GetRangeHeaderFromResponser� r r� �CREATEDr� rX r� r. �seekrW r� �RESUME_INCOMPLETE�_Upload__GetLastByter r� r� )r- Zrefresh_requestZrefresh_response� range_headerr r r r� X s2
��
� z"Upload.RefreshResumableUploadStatec C s |j �d|j �d��S )Nr� r )r r� r� r r r r z s z"Upload._GetRangeHeaderFromResponsec C s� | j du rt�d��|du r.|du r.t�d��| j tkr<dS |pD|j}|dur\|�|j�|_| �� tj ||| j
d�}|j t j
kr�tj�|��|j�d�| _|jd }|dur�|�|�}| �||� | jr�| �� S |S )z3Initialize this upload from the given http_request.Nz6No upload strategy set; did you call ConfigureRequest?r� )r� z X-Goog-Upload-Chunk-Granularity�location)r� r r� r r0 rx r= r> r r� r( r� r r� r� r� r r� r� r@ r+ r� )r- r� r0 r{ �
http_responser= r r r �InitializeUpload} s6
�
� �
zUpload.InitializeUploadc C s |� d�\}}}t|�S )N�-)r r� )r- r! r� r� r r r Z
__GetLastByte� s zUpload.__GetLastBytec C s6 | j d u rd S |p| j}|| j r2t�d| j ��d S )Nz0Server requires chunksize to be a multiple of %d)r� r, r �ConfigurationValueError)r- r, r r r Z__ValidateChunksize� s
��zUpload.__ValidateChunksizec C s |j dkp|j tjkp|jS )Ni� )r� r �TOO_MANY_REQUESTS� retry_afterr� r r r Z
__IsRetryable� s
��zUpload.__IsRetryablec s� �j tkrt�d��|p �j}|p&�j}�j}� �fdd�}� �fdd�}|rR|n|}|sj�jrjt�d��|rz���j � ��
� �j �sH|�j �
� �}|jtjtjfv r�d�_�qH|jtjtjtjfv�r�j tks��|�s�tj�|����� ��||� q�����|���_ �j d �j �
� k�r:t� d �j ����||� q��j�r�t �j d
��r��j �
� } �j � d t!j"� �j �
� }
�j � | � | |
k�r�t�#d t$|
�t$| � ����||� |S )
z1Helper function for StreamMedia / StreamInChunks.z"Cannot stream non-resumable uploadc s �j | � d�S �Nr� )�_Upload__SendChunk�rN �r� r- r r �
CallSendChunk� s �z+Upload.__StreamMedia.<locals>.CallSendChunkc s �j | � d�S r) )�_Upload__SendMediaBodyr+ r, r r �CallSendMediaBody� s �z/Upload.__StreamMedia.<locals>.CallSendMediaBodyz%Cannot gzip encode non-chunked uploadTr� z?Failed to transfer all bytes in chunk, upload paused at byte %dr r z7Upload complete with %s additional bytes left in stream)%r� r r rj r_ r` r� r� �_Upload__ValidateChunksizer, rG r� r. �tellr� r r� r r� r r �_Upload__IsRetryabler� r� r� rP r r r� rW �CommunicationError�hasattrr rf �SEEK_ENDrF r� ) r- rO r` r� r� r r- r/ Z send_funcZ current_posZend_posr r, r Z
__StreamMedia� sp
�
� �
� ���
�� zUpload.__StreamMediac C s | j |||dd�S )a� Send this resumable upload in a single request.
Args:
callback: Progress callback function with inputs
(http_wrapper.Response, transfer.Upload)
finish_callback: Final callback function with inputs
(http_wrapper.Response, transfer.Upload)
additional_headers: Dict of headers to include with the upload
http_wrapper.Request.
Returns:
http_wrapper.Response of final response.
Fr� ��_Upload__StreamMediar� r r r r� � s �zUpload.StreamMediac C s | j |||d�S )z'Send this (resumable) upload in chunks.)rO r` r� r6 r� r r r r� s �zUpload.StreamInChunksc C s^ dd� }t j| j|| j| j|d�}|jt jkrZ| �| �|��}|d |krZ| j �
|d � |S )z6Request helper function for SendMediaBody & SendChunk.c S s | d u rt �d| j ��d S )Nz,Request to url %s did not return a response.)r � RequestErrorr� )r r r r �
CheckResponse
s ��z0Upload.__SendMediaRequest.<locals>.CheckResponse)r* r� �check_response_funcr� ) r r� r6 r* r( r� r r r r. r )r- r� r� r9 r Z last_byter r r Z__SendMediaRequest s
� � zUpload.__SendMediaRequestc C s� | � � | jdu r t�d��t�| j| j| �}tj| j d|d�}| j
|j d< || jkrdd| j }nd|| jd | jf }||j d < |r�|j � |� | �
|| j�S )
z1Send the entire media stream in a single request.Nz*Total size must be known for SendMediaBodyr )r= r r� � Content-Type�
bytes */%s�bytes %s-%s/%sr� r )rG rX r rF r � StreamSlicer. r r� r= r� r� r� �_Upload__SendMediaRequest)r- rN r� � body_streamr� � range_stringr r r Z__SendMediaBody s*
� �
�
�
zUpload.__SendMediaBodyc
C sP | � � | jdu }tj| jdd�}| jrfd|jd< t�| j | j
�\}}}|| }| jdu r�|r�|| _ n`| jdu r�t �
| j || j
�}|j}|jr�|| _ |�| j
�}n$t|| j
| j�}t�| j || �}||_| j|jd< |r�d|_| jdu r�d||d f } n*||k�rd
| j } nd ||d | jf } | |jd < |�rD|j�|� | �||�S )
zSend the specified chunk.Nr )r= r r� r� r; r� z
bytes %s-%s/*r� r< r= r )rG rX r r� r= r� r� r r� r. r, r� r ZBufferedStreamZstream_end_positionZstream_exhaustedr� r� r r> r� r� r� r� r? )
r- rN r� Z no_log_bodyr� r@ Z read_lengthZ exhaustedr� rA r r r Z __SendChunk7 sD
�
�
zUpload.__SendChunk)NNFNTNNF)NTF)NTF)NFN)NN)N)NNNT)NNN)NNN)N)N)&rC rQ rR rS ru rt r1 rT rW r� rn ro r~ r r� r� r� r� rU rX r� r� r� r� r� r� r r$ r r0 r2 r7 r� r� r? r. r* r� r r rb r r ? sv �
� �
� $
-*"
�
B �
�
r )+rS �
__future__r �email.generator� generatorr �email.mime.multipart�mimer� r �email.mime.nonmultipart� nonmultipartr r� rr r� rf rL r: � six.movesr �apitools.base.pyr r r r r r �__all__r� r
r r r r r �objectr r
r �BytesGeneratorr� r r r r r �<module> s@ k q