Graphs and Sessions in Detail

What is a tf.Graph?

A tf.Graph object contains two relevant kinds of information$\colon$

• Graph structure.
• Graph collections$\colon$ collection of metadata stored in a tf.Graph object, two function tf.add_to_collection and tf.get_collection allow you store and access all object associated with a key.

Naming operations

A tf.Graph object defines a namespace for tf.Operation objects it contains. Tensorflow automatically chooses a unique name for each operation in your graph, but giving operations descriptive names can make your program easier to read and debug. Two ways are provided to override the name of an operation.

1. Each new tf.Operation or new tf.Tensor accepts an optional name argument.
2. The tf.name_scope function add a name scope prefix to all operation in a particular context.

prefix syntax is "<name_scopeA>/<name_scopeB>/.../<op-name>.

If a name_scope has already been used in the current context, TensorFlow appends "_1","_2"`, and so on.

The graph visualizer uses name scopes to group operations and reduce visual complexity of a graph.

tf.Tensor objects are implicitly named after the tf.Operation that produces the tnesor as output. A tensor name has the form "<OP_NAME>:<i>". "<OP_NAME>" is name of the operation. "<i>" is an integer representing the index of that tensor among the operation's outputs.

Executing a graph in a tf.Session

The tf.Session class is used to represnet a connection between the client program (typically a Python program) and the C++ runtime.

A tf.Session object provides access to devices in the local machine and remote devices using distributed Tensorflow runtime.

A tf.Session object also caches information about current tf.Graph object so that you can efficiently run the same computation multiple times.

A tf.Session is typically used as a context manager (in a with block) that automatically closes the session when you exit the block. It is also possible to create a session without using a with block, but tf.Session.close should be explicitly called when session is finished.

tf.Session.init

tf.Session.init function accepts three optional arguments$\colon$

target$\colon$ By default, target is left empty, then the session will only use device in the local machine.

the session access to all devices on a tensorflow server controlled machines by specify grpc://<URL> in target argument. <URL> is the address of the Tensorflow server.

graph$\colon$ By default, a new tf.Session object will be bound only to the current default graph. If you are using mulitple graphs in your program, you can specify an expilicit tf.Graph when you construct the session.

config$\colon$ config argument specify a tf.ConfigProto that controls the behavior of the session. Some of frequently used configuration options include$\colon$

allow_soft_placement enable a "soft" device placement algorithm, which ignores tf.device annotaions that attempt to place CPU-only operations on a GPU device.

cluster_def spcify what machines to use in the computation, and provide a mapping between job nmaes, task indices, and network address when using distributed TesnorFlow.

graph_options.optimizer_options provides control over the optimization on grpah before execution.

gpu_options.allow_growth increases the GPU memory allocator gradually rather than allocating most of the memory at startup.

tf.Session.run

The tf.Session.run function is the main mechanism for running a tf.Operation or evaluating a tf.Tensor by passing one or more tf.Operation or tf.Tensor objects to tf.Session.run.

syntax$\colon$

with tf.Session() as sess:
    <list of output> = sess.run(<list of fetches>, feed_dict=<dictinoary of feeds>)

tf.Session.run requires a list of fetches, which determine the return values. Fetches may be a tf.Operation, a tf.Tensor, or a tensor-like type1 such astf.Variable. These fetches determine what **subgraph** of the overalltf.Graph` must be executed to produce the result. This subgraph contains all operations named in the fetch list, plus all operations whose outputs are used to compute the value of the fetches.

tf.Session.run also optionally takes a dictionary of feeds, which is a mapping from tf.Tensor objects (typically tf.placeholder tensors) to values (typically Python scalars, lists, or Numpy arrays).

tf.Session.run also accepts an optional options argument to specify options about the call and optional run_metadata argument to collect metadata about the execution.

Programming with multiple graphs

[Common scenario]$\colon$ When training a model, a common way of organizing your code is to use one graph for training your model, and separate graph for evaluating or performing inference with a trained model. In many cases, the inference graph will be different from the training graph. When programming this way, you can either use completely separate Python processes to build and execute graphs, or you can use multiple graphs in the same process(detail in following article).

syntax$\colon$

g_1 = tf.Graph()
with g_1.as_default():
  ...
  sess_1 = tf.Session
g_2 = tf.Graph()
with g_2.as_default():
  ...
sess_2 =tf.Session(graph=g_2)

[0] https://www.tensorflow.org/guide/graphs