Shotgun Asynchronous Data Retriever

Introduction

The Shotgun data retriever makes it easy to pull in shotgun data and thumbnails in a background thread. It also manages caching of thumbnails on disk so that they don’t need to be retrieved over and over again.

You start the worker thread, then you can submit a series of requests which will be handled by the data object. Each time data arrives, a signal is emitted with the details of the data. Each object will by default have its own Shotgun API connection. Requests are prioritized so according to their priority. For example, Shotgun find() queries will always take precedence over shotgun thumbnail downloads.

Sample Code

The sample code below shows how you can use the data retriever in your Toolkit App Code:

# import the module - note that this is using the special
# import_framework code so it won't work outside an app
sg_data = sgtk.platform.import_framework("tk-framework-shotgunutils", "shotgun_data")

class ExampleWidget(QtGui.QWidget):

    def __init__(self):

        QtGui.QWidget.__init__(self)

        self.ui = Ui_Dialog()
        self.ui.setupUi(self)

        # set up data retriever
        self.__sg_data = sg_data.ShotgunDataRetriever(self)
        self.__sg_data.work_completed.connect( self.__on_worker_signal)
        self.__sg_data.work_failure.connect( self.__on_worker_failure)

        # and start its thread!
        self.__sg_data.start()

        # do an async request
        self._find_uid = self.__sg_data.execute_find("Shot", [], ["code"])


    def closeEvent(self, event):
        """
        Executed when the widget dialog is closed.
        """
        # gracefully stop our data retriever. This call
        # will block util the currently processing request has completed.
        self.__sg_data.stop()

        # okay to close dialog
        event.accept()

    def __on_worker_failure(self, uid, msg):
        """
        Asynchronous callback - the worker thread errored.
        """
        print "Error: %s" % msg

    def __on_worker_signal(self, uid, request_type, data):
        """
        Signaled whenever the worker completes something.
        """
        print "Data arrived: %s" % data

Class ShotgunDataRetriever

Note

Import the module into your Toolkit App using the following statement:

shotgun_data = sgtk.platform.import_framework("tk-framework-shotgunutils", "shotgun_data")
class shotgun_data.ShotgunDataRetriever(parent=None, sg=None, bg_task_manager=None)[source]

Asynchronous data retriever class which can be used to retrieve data and thumbnails from Shotgun and from disk thumbnail cache. Uses the BackgroundTaskManager to run tasks in background threads and emits signals when each query has either completed or failed. Requests are queued up using for example the execute_find() and request_thumbnail() methods.

Requests are executed in the following priority order:

  • First any thumbnails that are already cached on disk are handled.

  • Next, shotgun find() queries are handled.

  • Lastly thumbnail downloads are handled.

The thread will emit work_completed and work_failure signals when tasks are completed (or fail). The clear() method will clear the current queue. The currently processing item will finish processing and may send out signals even after a clear. Make sure you call the stop() method prior to destruction in order for the system to gracefully shut down.

Signal work_completed(uid, request_type, data_dict):

Emitted every time a requested task has completed. uid is a unique id which matches the unique id returned by the corresponding request call. request_type is a string denoting the type of request this event is associated with. data_dict is a dictionary containing the payload of the request. It will be different depending on what type of request it is.

Signal work_failure(uid, error_message):

Emitted every time a requested task has failed. uid is a unique id which matches the unique id returned by the corresponding request call.

Parameters:

  • parent (QWidget) – Parent object

  • sg – Optional Shotgun API Instance

  • bg_task_manager – Optional Task manager

Class bg_task_manager:

BackgroundTaskManager

static download_thumbnail(url, bundle)[source]

Convenience and compatibility method for quick and easy synchrnous thumbnail download. This will retrieve a shotgun thumbnail given a url - if it already exists in the cache, a path to it will be returned instantly. If not, it will be downloaded from Shotgun, placed in the standard cache location on disk and its path will be returned.

This method returns the transcoded version of the thumbnail originally uploaded to Shotgun. The image returned will always be a fixed-sized jpeg. To retrieve the thumbnail file in its original format and resolution, use ShotgunDataRetriever.download_thumbnail_source() instead.

This is a helper method meant to make it easy to port over synchronous legacy code - for a better solution, we recommend using the thumbnail retrieval that runs in a background thread.

Because Shotgun thumbnail urls have an expiry time, make sure to only pass urls to this method that have been very recently retrieved via a Shotgun find call.

Parameters:

  • url – The thumbnail url string that is associated with this thumbnail. This is the field value as returned by a Shotgun query.

  • bundle – App, Framework or Engine object requesting the download.

Returns:

A path to the thumbnail on disk.

static download_thumbnail_source(entity_type, entity_id, bundle)[source]

Convenience and compatibility method for quick and easy synchronous thumbnail download. This will retrieve the source file for a thumbnail given a shotgun entity type and id. If the resolved thumbnail source file has already been cached, a path to it will be returned instantly. Otherwise, it will be downloaded from Shotgun and placed in the standard cache location on disk. The full path to cached thumbnail is returned.

This method returns the thumbnail file in the original format and resolution it was uploaded to Shotgun as, which should be considered arbitrary. To retrieve a transcoded fixed-size jpeg version of the thumbnail, use ShotgunDataRetriever.download_thumbnail() instead.

This is a helper method meant to make it easy to port over synchronous legacy code - for a better solution, we recommend using the thumbnail retrieval that runs in a background thread.

Parameters:

  • entity_type (str) – Shotgun entity type with which the thumb is associated.

  • entity_id (int) – Shotgun entity id with which the thumb is associated.

  • bundle – App, Framework or Engine object requesting the download.

Returns:

A path to the thumbnail on disk.

start()[source]

Start the retriever thread.

Raises:

TankError if there is no BackgroundTaskManager associated with this instance

stop()[source]

Gracefully stop the receiver.

Once stop() has been called, the object needs to be discarded. This is a blocking call. It will synchronously wait until any potential currently processing item has completed.

Note that once stopped the data retriever can’t be restarted as the handle to the BackgroundTaskManager instance is released.

clear()[source]

Clears the queue.

Any currently processing item will complete without interruption, and signals will be sent out for these items.

stop_work(task_id)[source]

Stop the specified task

Parameters:

task_id – The task to stop

get_schema(project_id=None)[source]

Execute the schema_read and schema_entity_read methods asynchronously

Parameters:

project_id – If specified, the schema listing returned will be constrained by the schema settings for the given project.

Returns:

A unique identifier representing this request. This identifier is also part of the payload sent via the work_completed and work_failure signals, making it possible to match them up.

execute_find(*args, **kwargs)[source]

Executes a Shotgun find query asynchronously.

This method takes the same parameters as the Shotgun find() call.

The query will be queued up and once processed, either a work_completed or work_failure signal will be emitted.

Parameters:

  • *args – args to be passed to the Shotgun find() call

  • **kwargs – Named parameters to be passed to the Shotgun find() call

Returns:

A unique identifier representing this request. This identifier is also part of the payload sent via the work_completed and work_failure signals, making it possible to match them up.

execute_find_one(*args, **kwargs)[source]

Executes a Shotgun find_one query asynchronously.

This method takes the same parameters as the Shotgun find_one() call.

The query will be queued up and once processed, either a work_completed or work_failure signal will be emitted.

Parameters:

  • *args – args to be passed to the Shotgun find_one() call

  • **kwargs – Named parameters to be passed to the Shotgun find_one() call

Returns:

A unique identifier representing this request. This identifier is also part of the payload sent via the work_completed and work_failure signals, making it possible to match them up.

execute_update(*args, **kwargs)[source]

Execute a Shotgun update call asynchronously

This method takes the same parameters as the Shotgun update() call.

The query will be queued up and once processed, either a work_completed or work_failure signal will be emitted.

Parameters:

  • *args – args to be passed to the Shotgun update() call

  • **kwargs – Named parameters to be passed to the Shotgun update() call

Returns:

A unique identifier representing this request. This identifier is also part of the payload sent via the work_completed and work_failure signals, making it possible to match them up.

execute_create(*args, **kwargs)[source]

Execute a Shotgun create call asynchronously

The query will be queued up and once processed, either a work_completed or work_failure signal will be emitted.

This method takes the same parameters as the Shotgun create() call.

Parameters:

  • *args – args to be passed to the Shotgun create() call

  • **kwargs – Named parameters to be passed to the Shotgun create() call

Returns:

A unique identifier representing this request. This identifier is also part of the payload sent via the work_completed and work_failure signals, making it possible to match them up.

execute_delete(*args, **kwargs)[source]

Execute a Shotgun delete call asynchronously

This method takes the same parameters as the Shotgun delete() call.

The query will be queued up and once processed, either a work_completed or work_failure signal will be emitted.

Parameters:

  • *args – args to be passed to the Shotgun delete() call

  • **kwargs – Named parameters to be passed to the Shotgun delete() call

Returns:

A unique identifier representing this request. This identifier is also part of the payload sent via the work_completed and work_failure signals, making it possible to match them up.

execute_method(method, *args, **kwargs)[source]

Executes a generic execution of a method asynchronously. This is pretty much a wrapper for executing a task through the BackgroundTaskManager.

The specified method will be called on the following form:

method(sg, data)

Where sg is a shotgun API instance. Data is typically a dictionary with specific data that the method needs. The query will be queued up and once processed, either a work_completed or work_failure signal will be emitted.

Parameters:

  • method – The method that should be executed.

  • *args – args to be passed to the method

  • **kwargs – Named parameters to be passed to the method

Returns:

A unique identifier representing this request. This identifier is also part of the payload sent via the work_completed and work_failure signals, making it possible to match them up.

execute_text_search(*args, **kwargs)[source]

Executes a Shotgun text_search query asynchronously.

See the python api documentation here:

https://github.com/shotgunsoftware/python-api/wiki

This method takes the same parameters as the Shotgun text_search() call.

The query will be queued up and once processed, either a work_completed or work_failure signal will be emitted.

Parameters:

  • *args – args to be passed to the Shotgun text_search() call

  • **kwargs – Named parameters to be passed to the Shotgun text_search() call

Returns:

A unique identifier representing this request. This identifier is also part of the payload sent via the work_completed and work_failure signals, making it possible to match them up.

execute_nav_expand(*args, **kwargs)[source]

Executes a Shotgun nav_expand query asynchronously.

See the python api documentation here:

https://github.com/shotgunsoftware/python-api/wiki

This method takes the same parameters as the Shotgun nav_expand() call.

The query will be queued up and once processed, either a work_completed or work_failure signal will be emitted.

Parameters:

  • *args – args to be passed to the Shotgun nav_expand() call

  • **kwargs – Named parameters to be passed to the Shotgun nav_expand() call

Returns:

A unique identifier representing this request. This identifier is also part of the payload sent via the work_completed and work_failure signals, making it possible to match them up.

execute_nav_search_string(*args, **kwargs)[source]

Executes a Shotgun nav_search_string query asynchronously.

See the python api documentation here:

https://github.com/shotgunsoftware/python-api/wiki

This method takes the same parameters as the Shotgun nav_search_string() call.

The query will be queued up and once processed, either a work_completed or work_failure signal will be emitted.

Parameters:

  • *args – args to be passed to the Shotgun nav_search_string() call

  • **kwargs – Named parameters to be passed to the Shotgun nav_search_string() call

Returns:

A unique identifier representing this request. This identifier is also part of the payload sent via the work_completed and work_failure signals, making it possible to match them up.

execute_nav_search_entity(*args, **kwargs)[source]

Executes a Shotgun nav_search_entity query asynchronously.

See the python api documentation here:

https://github.com/shotgunsoftware/python-api/wiki

This method takes the same parameters as the Shotgun nav_search_entity() call.

The query will be queued up and once processed, either a work_completed or work_failure signal will be emitted.

Parameters:

  • *args – args to be passed to the Shotgun nav_search_entity() call

  • **kwargs – Named parameters to be passed to the Shotgun nav_search_entity() call

Returns:

A unique identifier representing this request. This identifier is also part of the payload sent via the work_completed and work_failure signals, making it possible to match them up.

request_attachment(attachment_entity)[source]

Downloads an attachment from Shotgun asynchronously or returns a cached file path if found.

Note

The provided Attachment entity definition must contain, at a minimum, the “this_file” substructure.

{
    "id": 597,
    "this_file": {
        "content_type": "image/png",
        "id": 597,
        "link_type": "upload",
        "name": "test.png",
        "type": "Attachment",
        "url": "https://abc.shotgunstudio.com/file_serve/attachment/597"
    },
    "type": "Attachment"
}
Parameters:

attachment_entity (dict) – The Attachment entity to download data from.

Returns:

A unique identifier representing this request.

request_thumbnail(url, entity_type, entity_id, field, load_image=False)[source]

Downloads a thumbnail from Shotgun asynchronously or returns a cached thumbnail if found. Optionally loads the thumbnail into a QImage.

Parameters:

  • url – The thumbnail url string that is associated with this thumbnail. This is the field value as returned by a Shotgun query.

  • entity_type – Shotgun entity type with which the thumb is associated.

  • entity_id – Shotgun entity id with which the thumb is associated.

  • field – Thumbnail field. Normally ‘image’ but could also for example be a deep link field such as sg_sequence.Sequence.image

  • load_image – If set to True, the return data structure will contain a QImage object with the image data loaded.

Returns:

A unique identifier representing this request. This identifier is also part of the payload sent via the work_completed and work_failure signals, making it possible to match them up.

request_thumbnail_source(entity_type, entity_id, load_image=False)[source]

Downloads a thumbnail from Shotgun asynchronously or returns a cached thumbnail if found. Optionally loads the thumbnail into a QImage.

Parameters:

  • entity_type – Shotgun entity type with which the thumb is associated.

  • entity_id – Shotgun entity id with which the thumb is associated.

  • load_image – If set to True, the return data structure will contain a QImage object with the image data loaded.

Returns:

A unique identifier representing this request. This identifier is also part of the payload sent via the work_completed and work_failure signals, making it possible to match them up.

blockSignals(self, b: bool) → bool
childEvent(self, event: PySide2.QtCore.QChildEvent) → None
children(self) → List[PySide2.QtCore.QObject]
static connect(arg__1: PySide2.QtCore.QObject, arg__2: bytes, arg__3: Callable, type: PySide2.QtCore.Qt.ConnectionType = PySide2.QtCore.Qt.ConnectionType.AutoConnection) → bool
static connect(self, arg__1: bytes, arg__2: Callable, type: PySide2.QtCore.Qt.ConnectionType = PySide2.QtCore.Qt.ConnectionType.AutoConnection) → bool
static connect(self, arg__1: bytes, arg__2: PySide2.QtCore.QObject, arg__3: bytes, type: PySide2.QtCore.Qt.ConnectionType = PySide2.QtCore.Qt.ConnectionType.AutoConnection) → bool
static connect(self, sender: PySide2.QtCore.QObject, signal: bytes, member: bytes, type: PySide2.QtCore.Qt.ConnectionType = PySide2.QtCore.Qt.ConnectionType.AutoConnection) → PySide2.QtCore.QMetaObject.Connection
static connect(sender: PySide2.QtCore.QObject, signal: PySide2.QtCore.QMetaMethod, receiver: PySide2.QtCore.QObject, method: PySide2.QtCore.QMetaMethod, type: PySide2.QtCore.Qt.ConnectionType = PySide2.QtCore.Qt.ConnectionType.AutoConnection) → PySide2.QtCore.QMetaObject.Connection
static connect(sender: PySide2.QtCore.QObject, signal: bytes, receiver: PySide2.QtCore.QObject, member: bytes, type: PySide2.QtCore.Qt.ConnectionType = PySide2.QtCore.Qt.ConnectionType.AutoConnection) → PySide2.QtCore.QMetaObject.Connection
connectNotify(self, signal: PySide2.QtCore.QMetaMethod) → None
customEvent(self, event: PySide2.QtCore.QEvent) → None
deleteLater(self) → None
static disconnect(arg__1: PySide2.QtCore.QMetaObject.Connection) → bool
static disconnect(arg__1: PySide2.QtCore.QObject, arg__2: bytes, arg__3: Callable) → bool
static disconnect(self, arg__1: bytes, arg__2: Callable) → bool
static disconnect(self, receiver: PySide2.QtCore.QObject, member: Optional[bytes] = None) → bool
static disconnect(self, signal: bytes, receiver: PySide2.QtCore.QObject, member: bytes) → bool
static disconnect(sender: PySide2.QtCore.QObject, signal: PySide2.QtCore.QMetaMethod, receiver: PySide2.QtCore.QObject, member: PySide2.QtCore.QMetaMethod) → bool
static disconnect(sender: PySide2.QtCore.QObject, signal: bytes, receiver: PySide2.QtCore.QObject, member: bytes) → bool
disconnectNotify(self, signal: PySide2.QtCore.QMetaMethod) → None
dumpObjectInfo(self) → None
dumpObjectTree(self) → None
dynamicPropertyNames(self) → List[PySide2.QtCore.QByteArray]
emit(self, arg__1: bytes, *args: None) → bool
event(self, event: PySide2.QtCore.QEvent) → bool
eventFilter(self, watched: PySide2.QtCore.QObject, event: PySide2.QtCore.QEvent) → bool
findChild(self, arg__1: type, arg__2: str = '') → object
findChildren(self, arg__1: type, arg__2: PySide2.QtCore.QRegExp) → Iterable
findChildren(self, arg__1: type, arg__2: PySide2.QtCore.QRegularExpression) → Iterable
findChildren(self, arg__1: type, arg__2: str = '') → Iterable
inherits(self, classname: bytes) → bool
installEventFilter(self, filterObj: PySide2.QtCore.QObject) → None
isSignalConnected(self, signal: PySide2.QtCore.QMetaMethod) → bool
isWidgetType(self) → bool
isWindowType(self) → bool
killTimer(self, id: int) → None
metaObject(self) → PySide2.QtCore.QMetaObject
moveToThread(self, thread: PySide2.QtCore.QThread) → None
objectName(self) → str
parent(self) → PySide2.QtCore.QObject
property(self, name: bytes) → Any
receivers(self, signal: bytes) → int
static registerUserData() → int
removeEventFilter(self, obj: PySide2.QtCore.QObject) → None
sender(self) → PySide2.QtCore.QObject
senderSignalIndex(self) → int
setObjectName(self, name: str) → None
setParent(self, parent: PySide2.QtCore.QObject) → None
setProperty(self, name: bytes, value: Any) → bool
signalsBlocked(self) → bool
startTimer(self, interval: int, timerType: PySide2.QtCore.Qt.TimerType = PySide2.QtCore.Qt.TimerType.CoarseTimer) → int
thread(self) → PySide2.QtCore.QThread
timerEvent(self, event: PySide2.QtCore.QTimerEvent) → None
tr(self, arg__1: bytes, arg__2: bytes = b'', arg__3: int = -1) → str