#
ConsisIDTransformer3DModel

A Diffusion Transformer model for 3D data from ConsisID was introduced in Identity-Preserving Text-to-Video Generation by Frequency Decomposition by Peking University & University of Rochester & etc.

The model can be loaded with the following code snippet.

    from diffusers import ConsisIDTransformer3DModel

    transformer = ConsisIDTransformer3DModel.from_pretrained("BestWishYsh/ConsisID-preview", subfolder="transformer", torch_dtype=torch.bfloat16).to("cuda")

##
ConsisIDTransformer3DModel

###
class diffusers.ConsisIDTransformer3DModel

< source >

( num_attention_heads: int = 30 attention_head_dim: int = 64 in_channels: int = 16 out_channels: typing.Optional[int] = 16 flip_sin_to_cos: bool = True freq_shift: int = 0 time_embed_dim: int = 512 text_embed_dim: int = 4096 num_layers: int = 30 dropout: float = 0.0 attention_bias: bool = True sample_width: int = 90 sample_height: int = 60 sample_frames: int = 49 patch_size: int = 2 temporal_compression_ratio: int = 4 max_text_seq_length: int = 226 activation_fn: str = 'gelu-approximate' timestep_activation_fn: str = 'silu' norm_elementwise_affine: bool = True norm_eps: float = 1e-05 spatial_interpolation_scale: float = 1.875 temporal_interpolation_scale: float = 1.0 use_rotary_positional_embeddings: bool = False use_learned_positional_embeddings: bool = False is_train_face: bool = False is_kps: bool = False cross_attn_interval: int = 1 LFE_num_tokens: int = 32 LFE_output_dim: int = 768 LFE_heads: int = 12 local_face_scale: float = 1.0 )

Parameters

- num_attention_heads (`int`, defaults to `30`) — The number of heads to use for multi-head attention.
- attention_head_dim (`int`, defaults to `64`) — The number of channels in each head.
- in_channels (`int`, defaults to `16`) — The number of channels in the input.
- out_channels (`int`, optional, defaults to `16`) — The number of channels in the output.
- flip_sin_to_cos (`bool`, defaults to `True`) — Whether to flip the sin to cos in the time embedding.
- time_embed_dim (`int`, defaults to `512`) — Output dimension of timestep embeddings.
- text_embed_dim (`int`, defaults to `4096`) — Input dimension of text embeddings from the text encoder.
- num_layers (`int`, defaults to `30`) — The number of layers of Transformer blocks to use.
- dropout (`float`, defaults to `0.0`) — The dropout probability to use.
- attention_bias (`bool`, defaults to `True`) — Whether or not to use bias in the attention projection layers.
- sample_width (`int`, defaults to `90`) — The width of the input latents.
- sample_height (`int`, defaults to `60`) — The height of the input latents.
- sample_frames (`int`, defaults to `49`) — The number of frames in the input latents. Note that this parameter was incorrectly initialized to 49 instead of 13 because ConsisID processed 13 latent frames at once in its default and recommended settings, but cannot be changed to the correct value to ensure backwards compatibility. To create a transformer with K latent frames, the correct value to pass here would be: ((K - 1) * temporal_compression_ratio + 1).
- patch_size (`int`, defaults to `2`) — The size of the patches to use in the patch embedding layer.
- temporal_compression_ratio (`int`, defaults to `4`) — The compression ratio across the temporal dimension. See documentation for `sample_frames`.
- max_text_seq_length (`int`, defaults to `226`) — The maximum sequence length of the input text embeddings.
- activation_fn (`str`, defaults to `"gelu-approximate"`) — Activation function to use in feed-forward.
- timestep_activation_fn (`str`, defaults to `"silu"`) — Activation function to use when generating the timestep embeddings.
- norm_elementwise_affine (`bool`, defaults to `True`) — Whether or not to use elementwise affine in normalization layers.
- norm_eps (`float`, defaults to `1e-5`) — The epsilon value to use in normalization layers.
- spatial_interpolation_scale (`float`, defaults to `1.875`) — Scaling factor to apply in 3D positional embeddings across spatial dimensions.
- temporal_interpolation_scale (`float`, defaults to `1.0`) — Scaling factor to apply in 3D positional embeddings across temporal dimensions.
- is_train_face (`bool`, defaults to `False`) — Whether to use enable the identity-preserving module during the training process. When set to `True`, the model will focus on identity-preserving tasks.
- is_kps (`bool`, defaults to `False`) — Whether to enable keypoint for global facial extractor. If `True`, keypoints will be in the model.
- cross_attn_interval (`int`, defaults to `1`) — The interval between cross-attention layers in the Transformer architecture. A larger value may reduce the frequency of cross-attention computations, which can help reduce computational overhead.
- LFE_num_tokens (`int`, defaults to `32`) — The number of tokens to use in the Local Facial Extractor (LFE). This module is responsible for capturing high frequency representations of the face.
- LFE_output_dim (`int`, defaults to `768`) — The output dimension of the Local Facial Extractor (LFE) module. This dimension determines the size of the feature vectors produced by the LFE module.
- LFE_heads (`int`, defaults to `12`) — The number of attention heads used in the Local Facial Extractor (LFE) module. More heads may improve the ability to capture diverse features, but can also increase computational complexity.
- local_face_scale (`float`, defaults to `1.0`) — A scaling factor used to adjust the importance of local facial features in the model. This can influence how strongly the model focuses on high frequency face-related content.

A Transformer model for video-like data in ConsisID.

####
fuse_qkv_projections

< source >

( )

Enables fused QKV projections. For self-attention modules, all projection matrices (i.e., query, key, value) are fused. For cross-attention modules, key and value projection matrices are fused.

This API is 🧪 experimental.

####
set_attn_processor

< source >

( processor: typing.Union[diffusers.models.attention_processor.AttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor2_0, diffusers.models.attention_processor.JointAttnProcessor2_0, diffusers.models.attention_processor.PAGJointAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0, diffusers.models.attention_processor.FusedJointAttnProcessor2_0, diffusers.models.attention_processor.AllegroAttnProcessor2_0, diffusers.models.attention_processor.AuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.CogVideoXAttnProcessor2_0, diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0, diffusers.models.attention_processor.XFormersAttnAddedKVProcessor, diffusers.models.attention_processor.XFormersAttnProcessor, diffusers.models.attention_processor.XLAFlashAttnProcessor2_0, diffusers.models.attention_processor.AttnProcessorNPU, diffusers.models.attention_processor.AttnProcessor2_0, diffusers.models.attention_processor.MochiVaeAttnProcessor2_0, diffusers.models.attention_processor.StableAudioAttnProcessor2_0, diffusers.models.attention_processor.HunyuanAttnProcessor2_0, diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.LuminaAttnProcessor2_0, diffusers.models.attention_processor.MochiAttnProcessor2_0, diffusers.models.attention_processor.FusedAttnProcessor2_0, diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0, diffusers.models.attention_processor.SlicedAttnProcessor, diffusers.models.attention_processor.SlicedAttnAddedKVProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor2_0, diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor, diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.LoRAAttnProcessor, diffusers.models.attention_processor.LoRAAttnProcessor2_0, diffusers.models.attention_processor.LoRAXFormersAttnProcessor, diffusers.models.attention_processor.LoRAAttnAddedKVProcessor, typing.Dict[str, typing.Union[diffusers.models.attention_processor.AttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor2_0, diffusers.models.attention_processor.JointAttnProcessor2_0, diffusers.models.attention_processor.PAGJointAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0, diffusers.models.attention_processor.FusedJointAttnProcessor2_0, diffusers.models.attention_processor.AllegroAttnProcessor2_0, diffusers.models.attention_processor.AuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.CogVideoXAttnProcessor2_0, diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0, diffusers.models.attention_processor.XFormersAttnAddedKVProcessor, diffusers.models.attention_processor.XFormersAttnProcessor, diffusers.models.attention_processor.XLAFlashAttnProcessor2_0, diffusers.models.attention_processor.AttnProcessorNPU, diffusers.models.attention_processor.AttnProcessor2_0, diffusers.models.attention_processor.MochiVaeAttnProcessor2_0, diffusers.models.attention_processor.StableAudioAttnProcessor2_0, diffusers.models.attention_processor.HunyuanAttnProcessor2_0, diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.LuminaAttnProcessor2_0, diffusers.models.attention_processor.MochiAttnProcessor2_0, diffusers.models.attention_processor.FusedAttnProcessor2_0, diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0, diffusers.models.attention_processor.SlicedAttnProcessor, diffusers.models.attention_processor.SlicedAttnAddedKVProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor2_0, diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor, diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.LoRAAttnProcessor, diffusers.models.attention_processor.LoRAAttnProcessor2_0, diffusers.models.attention_processor.LoRAXFormersAttnProcessor, diffusers.models.attention_processor.LoRAAttnAddedKVProcessor]]] )

Parameters

- processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`) — The instantiated processor class or a dictionary of processor classes that will be set as the processor for all `Attention` layers.

  If `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors.

Sets the attention processor to use to compute attention.

####
unfuse_qkv_projections

< source >

( )

Disables the fused QKV projection if enabled.

This API is 🧪 experimental.

##
Transformer2DModelOutput

###
class diffusers.models.modeling_outputs.Transformer2DModelOutput

< source >

( sample: torch.Tensor )

Parameters

- sample (`torch.Tensor` of shape `(batch_size, num_channels, height, width)` or `(batch size, num_vector_embeds - 1, num_latent_pixels)` if Transformer2DModel is discrete) — The hidden states output conditioned on the `encoder_hidden_states` input. If discrete, returns probability distributions for the unnoised latent pixels.

The output of Transformer2DModel.

< > Update on GitHub