Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,9 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+Qwen-VL-Chat/SimSun.ttf filter=lfs diff=lfs merge=lfs -text
+stable-diffusion-xl-base-1.0/01.png filter=lfs diff=lfs merge=lfs -text
+stable-diffusion-xl-base-1.0/comparison.png filter=lfs diff=lfs merge=lfs -text
+stable-diffusion-xl-base-1.0/text_encoder_2/model.onnx_data filter=lfs diff=lfs merge=lfs -text
+stable-diffusion-xl-base-1.0/unet/model.onnx_data filter=lfs diff=lfs merge=lfs -text
+stable-diffusion-xl-base-1.0/unet/openvino_model.xml filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,335 @@
+Tencent is pleased to support the open source community by making Seed-X available. 
+
+Copyright (C) 2024 THL A29 Limited, a Tencent company.  All rights reserved. 
+
+Seed-X is licensed under the Apache License Version 2.0 except for the third-party components listed below.
+
+
+Terms of the Apache License Version 2.0:
+--------------------------------------------------------------------
+Apache License 
+
+Version 2.0, January 2004
+
+http://www.apache.org/licenses/ 
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+1. Definitions.
+
+"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.
+
+"Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
+
+"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
+
+"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.
+
+"Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
+
+"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
+
+"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).
+
+"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
+
+"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
+
+"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
+
+2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
+
+You must give any other recipients of the Work or Derivative Works a copy of this License; and 
+
+You must cause any modified files to carry prominent notices stating that You changed the files; and 
+
+You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and 
+
+If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. 
+
+You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 
+
+5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
+
+6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
+
+7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
+
+8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
+
+9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS
+
+
+
+Other dependencies and licenses:
+
+
+Open Source Software Licensed under the Apache License Version 2.0:
+--------------------------------------------------------------------
+1. transformers
+Copyright 2018- The Hugging Face team. All rights reserved.
+Source code of this software can be obtained from: https://github.com/huggingface/transformers/blob/v4.30.2/
+
+2. diffusers
+Copyright 2023 The HuggingFace Team. All rights reserved.
+Source code of this software can be obtained from: https://github.com/huggingface/diffusers/blob/v0.25.0/
+
+A copy of Apache 2.0 has been included in this file.
+
+
+
+Open Source Software Licensed under the BSD 3-Clause License:
+--------------------------------------------------------------------
+1. torchvision
+Copyright (c) Soumith Chintala 2016, 
+All rights reserved.
+
+Terms of the BSD 3-Clause License:
+--------------------------------------------------------------------
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
+
+Open Source Software Licensed under the BSD 3-Clause License and Other Licenses of the Third-Party Components therein:
+--------------------------------------------------------------------
+1. numpy
+Copyright (c) 2005-2021, NumPy Developers.
+All rights reserved.
+
+A copy of the BSD 3-Clause License is included in this file.
+
+For the license of other third party components, please refer to the following URL:
+https://github.com/numpy/numpy/blob/v1.20.1/LICENSES_bundled.txt
+
+
+
+Open Source Software Licensed under the BSD 3-Clause License and Other Licenses of the Third-Party Components therein:
+--------------------------------------------------------------------
+1. torch
+Copyright (c) 2016-     Facebook, Inc            (Adam Paszke)
+Copyright (c) 2014-     Facebook, Inc            (Soumith Chintala)
+Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
+Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu)
+Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
+Copyright (c) 2011-2013 NYU                      (Clement Farabet)
+Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
+Copyright (c) 2006      Idiap Research Institute (Samy Bengio)
+Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
+
+A copy of the BSD 3-Clause License is included in this file.
+
+For the license of other third party components, please refer to the following URL:
+https://github.com/pytorch/pytorch/blob/v2.0.1/NOTICE
+
+
+
+Open Source Software Licensed under the LLAMA 2 Community License:
+--------------------------------------------------------------------
+1. Llama 2
+Copyright (c) Meta Platforms, Inc. All Rights Reserved.
+
+
+Terms of the LLAMA 2 COMMUNITY LICENSE AGREEMENT:
+--------------------------------------------------------------------
+LLAMA 2 COMMUNITY LICENSE AGREEMENT	
+Llama 2 Version Release Date: July 18, 2023
+
+"Agreement" means the terms and conditions for use, reproduction, distribution and 
+modification of the Llama Materials set forth herein.
+
+"Documentation" means the specifications, manuals and documentation 
+accompanying Llama 2 distributed by Meta at ai.meta.com/resources/models-and-
+libraries/llama-downloads/.
+
+"Licensee" or "you" means you, or your employer or any other person or entity (if 
+you are entering into this Agreement on such person or entity's behalf), of the age 
+required under applicable laws, rules or regulations to provide legal consent and that 
+has legal authority to bind your employer or such other person or entity if you are 
+entering in this Agreement on their behalf.
+
+"Llama 2" means the foundational large language models and software and 
+algorithms, including machine-learning model code, trained model weights, 
+inference-enabling code, training-enabling code, fine-tuning enabling code and other 
+elements of the foregoing distributed by Meta at ai.meta.com/resources/models-and-
+libraries/llama-downloads/.
+
+"Llama Materials" means, collectively, Meta's proprietary Llama 2 and 
+Documentation (and any portion thereof) made available under this Agreement.
+
+"Meta" or "we" means Meta Platforms Ireland Limited (if you are located in or, if you 
+are an entity, your principal place of business is in the EEA or Switzerland) and Meta 
+Platforms, Inc. (if you are located outside of the EEA or Switzerland). 
+
+By clicking "I Accept" below or by using or distributing any portion or element of the 
+Llama Materials, you agree to be bound by this Agreement.
+
+1. License Rights and Redistribution. 
+
+      a. Grant of Rights. You are granted a non-exclusive, worldwide, non-
+transferable and royalty-free limited license under Meta's intellectual property or 
+other rights owned by Meta embodied in the Llama Materials to use, reproduce, 
+distribute, copy, create derivative works of, and make modifications to the Llama 
+Materials.  
+      
+      b. Redistribution and Use.  
+
+            i. If you distribute or make the Llama Materials, or any derivative works 
+thereof, available to a third party, you shall provide a copy of this Agreement to such 
+third party. 
+            ii.  If you receive Llama Materials, or any derivative works thereof, from 
+a Licensee as part of an integrated end user product, then Section 2 of this 
+Agreement will not apply to you. 
+
+            iii. You must retain in all copies of the Llama Materials that you 
+distribute the following attribution notice within a "Notice" text file distributed as a 
+part of such copies: "Llama 2 is licensed under the LLAMA 2 Community License, 
+Copyright (c) Meta Platforms, Inc. All Rights Reserved."
+
+            iv. Your use of the Llama Materials must comply with applicable laws 
+and regulations (including trade compliance laws and regulations) and adhere to the 
+Acceptable Use Policy for the Llama Materials (available at 
+https://ai.meta.com/llama/use-policy), which is hereby incorporated by reference into 
+this Agreement.
+
+            v. You will not use the Llama Materials or any output or results of the 
+Llama Materials to improve any other large language model (excluding Llama 2 or 
+derivative works thereof).  
+
+2. Additional Commercial Terms. If, on the Llama 2 version release date, the 
+monthly active users of the products or services made available by or for Licensee, 
+or Licensee's affiliates, is greater than 700 million monthly active users in the 
+preceding calendar month, you must request a license from Meta, which Meta may 
+grant to you in its sole discretion, and you are not authorized to exercise any of the 
+rights under this Agreement unless or until Meta otherwise expressly grants you 
+such rights.
+            
+3. Disclaimer of Warranty. UNLESS REQUIRED BY APPLICABLE LAW, THE 
+LLAMA MATERIALS AND ANY OUTPUT AND RESULTS THEREFROM ARE 
+PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, 
+EITHER EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, ANY 
+WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, OR 
+FITNESS FOR A PARTICULAR PURPOSE. YOU ARE SOLELY RESPONSIBLE 
+FOR DETERMINING THE APPROPRIATENESS OF USING OR REDISTRIBUTING 
+THE LLAMA MATERIALS AND ASSUME ANY RISKS ASSOCIATED WITH YOUR 
+USE OF THE LLAMA MATERIALS AND ANY OUTPUT AND RESULTS.
+
+4. Limitation of Liability. IN NO EVENT WILL META OR ITS AFFILIATES BE 
+LIABLE UNDER ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, TORT, 
+NEGLIGENCE, PRODUCTS LIABILITY, OR OTHERWISE, ARISING OUT OF THIS 
+AGREEMENT, FOR ANY LOST PROFITS OR ANY INDIRECT, SPECIAL, 
+CONSEQUENTIAL, INCIDENTAL, EXEMPLARY OR PUNITIVE DAMAGES, EVEN 
+IF META OR ITS AFFILIATES HAVE BEEN ADVISED OF THE POSSIBILITY OF 
+ANY OF THE FOREGOING.
+ 
+5. Intellectual Property.
+
+      a. No trademark licenses are granted under this Agreement, and in 
+connection with the Llama Materials, neither Meta nor Licensee may use any name 
+or mark owned by or associated with the other or any of its affiliates, except as 
+required for reasonable and customary use in describing and redistributing the 
+Llama Materials.
+
+      b. Subject to Meta's ownership of Llama Materials and derivatives made by or 
+for Meta, with respect to any derivative works and modifications of the Llama 
+Materials that are made by you, as between you and Meta, you are and will be the 
+owner of such derivative works and modifications.
+
+      c. If you institute litigation or other proceedings against Meta or any entity 
+(including a cross-claim or counterclaim in a lawsuit) alleging that the Llama 
+Materials or Llama 2 outputs or results, or any portion of any of the foregoing, 
+constitutes infringement of intellectual property or other rights owned or licensable 
+by you, then any licenses granted to you under this Agreement shall terminate as of 
+the date such litigation or claim is filed or instituted. You will indemnify and hold 
+harmless Meta from and against any claim by any third party arising out of or related 
+to your use or distribution of the Llama Materials.
+
+6. Term and Termination. The term of this Agreement will commence upon your 
+acceptance of this Agreement or access to the Llama Materials and will continue in 
+full force and effect until terminated in accordance with the terms and conditions 
+herein. Meta may terminate this Agreement if you are in breach of any term or 
+condition of this Agreement. Upon termination of this Agreement, you shall delete 
+and cease use of the Llama Materials. Sections 3, 4 and 7 shall survive the 
+termination of this Agreement. 
+
+7. Governing Law and Jurisdiction. This Agreement will be governed and 
+construed under the laws of the State of California without regard to choice of law 
+principles, and the UN Convention on Contracts for the International Sale of Goods 
+does not apply to this Agreement. The courts of California shall have exclusive 
+jurisdiction of any dispute arising out of this Agreement. 
+
+
+
+Open Source Software Licensed under the Tongyi Qianwen LICENSE AGREEMENT:
+--------------------------------------------------------------------
+1. Qwen-VL
+Copyright (c) Alibaba Cloud. All Rights Reserved.
+
+
+Terms of the Tongyi Qianwen LICENSE AGREEMENT:
+--------------------------------------------------------------------
+Tongyi Qianwen LICENSE AGREEMENT
+
+Tongyi Qianwen Release Date: August 23, 2023
+
+By clicking to agree or by using or distributing any portion or element of the Tongyi Qianwen Materials, you will be deemed to have recognized and accepted the content of this Agreement, which is effective immediately.
+
+1. Definitions
+    a. This Tongyi Qianwen LICENSE AGREEMENT (this "Agreement") shall mean the terms and conditions for use, reproduction, distribution and modification of the Materials as defined by this Agreement.
+    b. "We"(or "Us") shall mean Alibaba Cloud.
+    c. "You" (or "Your") shall mean a natural person or legal entity exercising the rights granted by this Agreement and/or using the Materials for any purpose and in any field of use.
+    d. "Third Parties" shall mean individuals or legal entities that are not under common control with Us or You.
+    e. "Tongyi Qianwen" shall mean the large language models (including Qwen-VL model and Qwen-VL-Chat model), and software and algorithms, consisting of trained model weights, parameters (including optimizer states), machine-learning model code, inference-enabling code, training-enabling code, fine-tuning enabling code and other elements of the foregoing distributed by Us.
+    f. "Materials" shall mean, collectively, Alibaba Cloud's proprietary Tongyi Qianwen and Documentation (and any portion thereof) made available under this Agreement.
+    g. "Source" form shall mean the preferred form for making modifications, including but not limited to model source code, documentation source, and configuration files.
+    h. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+2. Grant of Rights
+You are granted a non-exclusive, worldwide, non-transferable and royalty-free limited license under Alibaba Cloud's intellectual property or other rights owned by Us embodied in the Materials to use, reproduce, distribute, copy, create derivative works of, and make modifications to the Materials.
+
+3. Redistribution
+You may reproduce and distribute copies of the Materials or derivative works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
+    a. You shall give any other recipients of the Materials or derivative works a copy of this Agreement;
+    b. You shall cause any modified files to carry prominent notices stating that You changed the files;
+    c. You shall retain in all copies of the Materials that You distribute the following attribution notices within a "Notice" text file distributed as a part of such copies: "Tongyi Qianwen is licensed under the Tongyi Qianwen LICENSE AGREEMENT, Copyright (c) Alibaba Cloud. All Rights Reserved."; and
+    d. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such derivative works as a whole, provided Your use, reproduction, and distribution of the work otherwise complies with the terms and conditions of this Agreement.
+
+4. Restrictions
+If you are commercially using the Materials, and your product or service has more than 100 million monthly active users, You shall request a license from Us. You cannot exercise your rights under this Agreement without our express authorization.
+
+5. Rules of use
+    a. The Materials may be subject to export controls or restrictions in China, the United States or other countries or regions. You shall comply with applicable laws and regulations in your use of the Materials.
+    b. You can not use the Materials or any output therefrom to improve any other large language model (excluding Tongyi Qianwen or derivative works thereof).
+
+6. Intellectual Property
+    a. We retain ownership of all intellectual property rights in and to the Materials and derivatives made by or for Us. Conditioned upon compliance with the terms and conditions of this Agreement, with respect to any derivative works and modifications of the Materials that are made by you, you are and will be the owner of such derivative works and modifications.
+    b. No trademark license is granted to use the trade names, trademarks, service marks, or product names of Us, except as required to fulfill notice requirements under this Agreement or as required for reasonable and customary use in describing and redistributing the Materials.
+    c. If you commence a lawsuit or other proceedings (including a cross-claim or counterclaim in a lawsuit) against Us or any entity alleging that the Materials or any output therefrom, or any part of the foregoing, infringe any intellectual property or other right owned or licensable by you, then all licences granted to you under this Agreement shall terminate as of the date such lawsuit or other proceeding is commenced or brought.
+
+7. Disclaimer of Warranty and Limitation of Liability
+
+    a. We are not obligated to support, update, provide training for, or develop any further version of the Tongyi Qianwen Materials or to grant any license thereto.
+    b. THE MATERIALS ARE PROVIDED "AS IS" WITHOUT ANY EXPRESS OR IMPLIED WARRANTY OF ANY KIND INCLUDING WARRANTIES OF MERCHANTABILITY, NONINFRINGEMENT, OR FITNESS FOR A PARTICULAR PURPOSE. WE MAKE NO WARRANTY AND ASSUME NO RESPONSIBILITY FOR THE SAFETY OR STABILITY OF THE MATERIALS AND ANY OUTPUT THEREFROM.
+    c. IN NO EVENT SHALL WE BE LIABLE TO YOU FOR ANY DAMAGES, INCLUDING, BUT NOT LIMITED TO ANY DIRECT, OR INDIRECT, SPECIAL OR CONSEQUENTIAL DAMAGES ARISING FROM YOUR USE OR INABILITY TO USE THE MATERIALS OR ANY OUTPUT OF IT, NO MATTER HOW IT’S CAUSED.
+    d. You will defend, indemnify and hold harmless Us from and against any claim by any third party arising out of or related to your use or distribution of the Materials.
+
+8. Survival and Termination.
+    a. The term of this Agreement shall commence upon your acceptance of this Agreement or access to the Materials and will continue in full force and effect until terminated in accordance with the terms and conditions herein.
+    b. We may terminate this Agreement if you breach any of the terms or conditions of this Agreement. Upon termination of this Agreement, you must delete and cease use of the Materials. Sections 7 and 9 shall survive the termination of this Agreement.
+
+9. Governing Law and Jurisdiction.
+    a. This Agreement and any dispute arising out of or relating to it will be governed by the laws of China, without regard to conflict of law principles, and the UN Convention on Contracts for the International Sale of Goods does not apply to this Agreement.
+    b. The People's Courts in Hangzhou City shall have exclusive jurisdiction over any dispute arising out of this Agreement.

Qwen-VL-Chat/.gitattributes

@@ -0,0 +1,37 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+SimSun.ttf filter=lfs diff=lfs merge=lfs -text
+assets/apple.jpeg filter=lfs diff=lfs merge=lfs -text

Qwen-VL-Chat/README.md

@@ -0,0 +1,727 @@
+---
+language:
+- zh
+- en
+tags:
+- qwen
+pipeline_tag: text-generation
+inference: false
+---
+
+# Qwen-VL-Chat
+
+<br>
+
+<p align="center">
+    <img src="https://qianwen-res.oss-cn-beijing.aliyuncs.com/logo_vl.jpg" width="400"/>
+<p>
+<br>
+
+<p align="center">
+  Qwen-VL 
+  <a href="https://huggingface.co/Qwen/Qwen-VL">🤗</a>
+  <a href="https://modelscope.cn/models/qwen/Qwen-VL/summary">🤖</a>&nbsp ｜ 
+  Qwen-VL-Chat 
+  <a href="https://huggingface.co/Qwen/Qwen-VL-Chat">🤗</a>
+  <a href="https://modelscope.cn/models/qwen/Qwen-VL-Chat/summary">🤖</a>&nbsp 
+  (Int4: 
+  <a href="https://huggingface.co/Qwen/Qwen-VL-Chat-Int4">🤗</a> 
+  <a href="https://modelscope.cn/models/qwen/Qwen-VL-Chat-Int4/summary">🤖</a>&nbsp) ｜
+  Qwen-VL-Plus 
+  <a href="https://huggingface.co/spaces/Qwen/Qwen-VL-Plus">🤗</a> 
+  <a href="https://modelscope.cn/studios/qwen/Qwen-VL-Chat-Demo/summary">🤖</a>&nbsp ｜ 
+  Qwen-VL-Max 
+  <a href="https://huggingface.co/spaces/Qwen/Qwen-VL-Max">🤗</a>
+  <a href="https://modelscope.cn/studios/qwen/Qwen-VL-Max/summary">🤖</a>&nbsp
+<br>
+  <a href="https://tongyi.aliyun.com/qianwen">Web</a>&nbsp&nbsp | &nbsp&nbsp
+  <a href="https://help.aliyun.com/zh/dashscope/developer-reference/vl-plus-quick-start">API</a>&nbsp&nbsp | &nbsp&nbsp
+  <a href="assets/wechat.png">WeChat</a>&nbsp&nbsp | &nbsp&nbsp
+  <a href="https://discord.gg/z3GAxXZ9Ce">Discord</a>&nbsp&nbsp | &nbsp&nbsp
+  <a href="https://arxiv.org/abs/2308.12966">Paper</a>&nbsp&nbsp | &nbsp&nbsp
+  <a href="TUTORIAL.md">Tutorial</a>
+</p>
+<br>
+
+**Qwen-VL** 是阿里云研发的大规模视觉语言模型（Large Vision Language Model, LVLM）。Qwen-VL 可以以图像、文本、检测框作为输入，并以文本和检测框作为输出。Qwen-VL 系列模型性能强大，具备多语言对话、多图交错对话等能力，并支持中文开放域定位和细粒度图像识别与理解。
+
+**Qwen-VL** (Qwen Large Vision Language Model) is the visual multimodal version of the large model series, Qwen (abbr. Tongyi Qianwen), proposed by Alibaba Cloud. Qwen-VL accepts image, text, and bounding box as inputs, outputs text and bounding box. The features of Qwen-VL include:
+
+目前，我们提供了Qwen-VL和Qwen-VL-Chat两个模型，分别为预训练模型和Chat模型。如果想了解更多关于模型的信息，请点击[链接](https://github.com/QwenLM/Qwen-VL/blob/master/visual_memo.md)查看我们的技术备忘录。本仓库为Qwen-VL-Chat仓库。
+
+We release Qwen-VL and Qwen-VL-Chat, which are pretrained model and Chat model respectively. For more details about Qwen-VL, please refer to our [technical memo](https://github.com/QwenLM/Qwen-VL/blob/master/visual_memo.md). This repo is the one for Qwen-VL-Chat.
+<br>
+
+## 安装要求 (Requirements)
+
+* python 3.8及以上版本
+* pytorch 1.12及以上版本，推荐2.0及以上版本
+* 建议使用CUDA 11.4及以上（GPU用户需考虑此选项）
+* python 3.8 and above
+* pytorch 1.12 and above, 2.0 and above are recommended
+* CUDA 11.4 and above are recommended (this is for GPU users)
+  <br>
+
+## 快速开始 (Quickstart)
+
+我们提供简单的示例来说明如何利用 🤗 Transformers 快速使用Qwen-VL-Chat。
+
+在开始前，请确保你已经配置好环境并安装好相关的代码包。最重要的是，确保你满足上述要求，然后安装相关的依赖库。
+
+Below, we provide simple examples to show how to use Qwen-VL-Chat with 🤗 Transformers.
+
+Before running the code, make sure you have setup the environment and installed the required packages. Make sure you meet the above requirements, and then install the dependent libraries.
+
+```bash
+pip install -r requirements.txt
+```
+
+接下来你可以开始使用Transformers来使用我们的模型。关于视觉模块的更多用法，请参考[教程](TUTORIAL.md)。
+
+Now you can start with Transformers. More usage aboue vision encoder, please refer to [tutorial](TUTORIAL_zh.md).
+
+#### 🤗 Transformers
+
+To use Qwen-VL-Chat for the inference, all you need to do is to input a few lines of codes as demonstrated below. However, **please make sure that you are using the latest code.**
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from transformers.generation import GenerationConfig
+import torch
+torch.manual_seed(1234)
+
+# Note: The default behavior now has injection attack prevention off.
+tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen-VL-Chat", trust_remote_code=True)
+
+# use bf16
+# model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-VL-Chat", device_map="auto", trust_remote_code=True, bf16=True).eval()
+# use fp16
+# model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-VL-Chat", device_map="auto", trust_remote_code=True, fp16=True).eval()
+# use cpu only
+# model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-VL-Chat", device_map="cpu", trust_remote_code=True).eval()
+# use cuda device
+model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-VL-Chat", device_map="cuda", trust_remote_code=True).eval()
+
+# Specify hyperparameters for generation (No need to do this if you are using transformers>=4.32.0)
+# model.generation_config = GenerationConfig.from_pretrained("Qwen/Qwen-VL-Chat", trust_remote_code=True)
+
+# 1st dialogue turn
+query = tokenizer.from_list_format([
+    {'image': 'https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg'},
+    {'text': '这是什么'},
+])
+response, history = model.chat(tokenizer, query=query, history=None)
+print(response)
+# 图中是一名年轻女子在沙滩上和她的狗玩耍，狗的品种可能是拉布拉多。她们坐在沙滩上，狗的前腿抬起来，似乎在和人类击掌。两人之间充满了信任和爱。
+
+# 2nd dialogue turn
+response, history = model.chat(tokenizer, '输出"击掌"的检测框', history=history)
+print(response)
+# <ref>击掌</ref><box>(517,508),(589,611)</box>
+image = tokenizer.draw_bbox_on_latest_picture(response, history)
+if image:
+  image.save('1.jpg')
+else:
+  print("no box")
+```
+
+<p align="center">
+    <img src="https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo_highfive.jpg" width="500"/>
+<p>
+<br>
+
+## 量化 (Quantization)
+
+### 用法 (Usage)
+
+当前我们提供了基于[AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ)的量化方案，并提供了Qwen-VL-Chat的Int4量化版本Qwen-VL-Chat-Int4 [点击此处](https://huggingface.co/Qwen/Qwen-VL-Chat-Int4)。该模型在效果评测上几乎无损，并在显存占用和推理速度上具有明显优势。
+
+下文说明如何使用该量化模型。开始之前，请确保你满足要求（如torch2.0及以上、transformers 4.32.0及以上，等）并安装所需的代码库：
+
+We provide a new solution based on [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ), and release an Int4 quantized model for Qwen-VL-Chat, Qwen-VL-Chat-Int4 [Click here](https://huggingface.co/Qwen/Qwen-VL-Chat-Int4), which achieves nearly lossless model effects but improved performance on both memory costs and inference speed.
+
+Here we demonstrate how to use our provided quantized models for inference. Before you start, make sure you meet the requirements (e.g., torch 2.0 and above, transformers 4.32.0 and above, etc.) and install the required packages:
+
+```bash
+pip install optimum
+git clone https://github.com/JustinLin610/AutoGPTQ.git & cd AutoGPTQ
+pip install -v .
+```
+
+如遇到安装 `auto-gptq` 的问题，建议您前往官方[repo](https://github.com/PanQiWei/AutoGPTQ) 寻找合适的wheel。
+
+随后你便可以按照上述用法，轻松调用量化模型：
+
+If you meet problems installing `auto-gptq`, we advise you to check out the official [repo](https://github.com/PanQiWei/AutoGPTQ) to find a wheel.
+
+Then you can load the quantized model easily and run inference as same as usual:
+
+```python
+model = AutoModelForCausalLM.from_pretrained(
+    "Qwen/Qwen-VL-Chat-Int4",
+    device_map="auto",
+    trust_remote_code=True
+).eval()
+# Either a local path or an u[](https://)rl between <img></img> tags.
+image_path = 'https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg'
+response, history = model.chat(tokenizer, query=f'<img>{image_path}</img>这是什么', history=None)
+print(response)
+```
+
+### 效果评测 (Performance)
+
+我们列出不同精度下模型在评测基准 **[TouchStone](https://github.com/OFA-Sys/TouchStone)** 上的表现，并发现量化模型并没有显著性能损失。结果如下所示：
+
+We illustrate the model performance of both BF16 and Int4 models on the benchmark **[TouchStone](https://github.com/OFA-Sys/TouchStone)**, and we find that the quantized model does not suffer from significant performance degradation. Results are shown below:
+
+| Quantization | ZH.        | EN            |
+| ------------ | :--------: | :-----------: | 
+| BF16         | 401.2      |    645.2      |
+| Int4         | 386.6      |    651.4      |
+
+### 推理速度 (Inference Speed)
+
+我们测算了在输入一张图片（即258个token）的条件下BF16和Int4的模型生成1792 (2048-258) 和 7934 (8192-258) 个token的平均速度。
+
+We measured the average inference speed (tokens/s) of generating 1792 (2048-258) and 7934 (8192-258) tokens with the context of an image (which takes 258 tokens) under BF16 precision and Int4 quantization, respectively.
+
+| Quantization | Speed (2048 tokens) | Speed (8192 tokens) |
+| ------------ | :-----------------: | :-----------------: |
+| BF16         |        28.87        |        24.32        |
+| Int4         |        37.79        |        34.34        |
+
+推理速度测算是在单卡 A100-SXM4-80G GPU上运行，使用PyTorch 2.0.1及CUDA 11.4。
+
+The profiling runs on a single A100-SXM4-80G GPU with PyTorch 2.0.1 and CUDA 11.4.
+
+### GPU显存占用 (GPU Memory Usage)
+
+我们还测算了在一张图片输入的条件下BF16和Int4模型生成1792 (2048-258) 和 7934 (8192-258) 个token所需显存。结果如下所示：
+
+We also profile the peak GPU memory usage for encoding 1792 (2048-258) tokens (including an image) as context (and generating single token) and generating 7934 (8192-258) tokens (with an image as context) under BF16 or Int4 quantization level, respectively. The results are shown below.
+
+| Quantization | Peak Usage for Encoding 2048 Tokens | Peak Usage for Generating 8192 Tokens |
+| ------------ | :---------------------------------: | :-----------------------------------: |
+| BF16         |               22.60GB               |                28.01GB                |
+| Int4         |               11.82GB               |                17.23GB                |
+
+上述速度和显存测算使用[此脚本](https://qianwen-res.oss-cn-beijing.aliyuncs.com/profile_mm.py)完成。
+
+The above speed and memory profiling are conducted using [this script](https://qianwen-res.oss-cn-beijing.aliyuncs.com/profile_mm.py).
+<br>
+
+## 评测
+
+我们从两个角度评测了两个模型的能力：
+
+1. 在**英文标准 Benchmark** 上评测模型的基础任务能力。目前评测了四大类多模态任务：
+   
+   - Zero-shot Caption: 评测模型在未见过数据集上的零样本图片描述能力；
+   - General VQA: 评测模型的通用问答能力，例如判断题、颜色、个数、类目等问答能力；
+   - Text-based VQA：评测模型对于图片中文字相关的识别/问答能力，例如文档问答、图表问答、文字问答等；
+   - Referring Expression Compression：评测模型给定物体描述画检测框的能力；
+2. **试金石 (TouchStone)**：为了评测模型整体的图文对话能力和人类对齐水平。我们为此构建了一个基于 GPT4 打分来评测 LVLM 模型的 Benchmark：TouchStone。在 TouchStone-v0.1 中：
+   
+   - 评测基准总计涵盖 300+张图片、800+道题目、27个类别。包括基础属性问答、人物地标问答、影视作品问答、视觉推理、反事实推理、诗歌创作、故事写作，商品比较、图片解题等**尽可能广泛的类别**。
+   - 为了弥补目前 GPT4 无法直接读取图片的缺陷，我们给所有的带评测图片提供了**人工标注的充分详细描述**，并且将图片的详细描述、问题和模型的输出结果一起交给 GPT4 打分。
+   - 评测同时包含英文版本和中文版本。
+
+评测结果如下：
+
+We evaluated the model's ability from two perspectives:
+
+1. **Standard Benchmarks**: We evaluate the model's basic task capabilities on four major categories of multimodal tasks:
+   
+   - Zero-shot Caption: Evaluate model's zero-shot image captioning ability on unseen datasets;
+   - General VQA: Evaluate the general question-answering ability of pictures, such as the judgment, color, number, category, etc;
+   - Text-based VQA: Evaluate the model's ability to recognize text in pictures, such as document QA, chart QA, etc;
+   - Referring Expression Comprehension: Evaluate the ability to localize a target object in an image described by a referring expression.
+2. **TouchStone**: To evaluate the overall text-image dialogue capability and alignment level with humans, we have constructed a benchmark called TouchStone, which is based on scoring with GPT4 to evaluate the LVLM model.
+   
+   - The TouchStone benchmark covers a total of 300+ images, 800+ questions, and 27 categories. Such as attribute-based Q&A, celebrity recognition, writing poetry, summarizing multiple images, product comparison, math problem solving, etc;
+   - In order to break the current limitation of GPT4 in terms of direct image input, TouchStone provides fine-grained image annotations by human labeling. These detailed annotations, along with the questions and the model's output, are then presented to GPT4 for scoring.
+   - The benchmark includes both English and Chinese versions.
+
+The results of the evaluation are as follows:
+
+Qwen-VL outperforms current SOTA generalist models on multiple VL tasks and has a more comprehensive coverage in terms of capability range.
+
+<p align="center">
+    <img src="https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/radar.png" width="600"/>
+<p>
+
+### 零样本图像描述 & 通用视觉问答 (Zero-shot Captioning & General VQA)
+
+<table>
+<thead>
+  <tr>
+    <th rowspan="2">Model type</th>
+    <th rowspan="2">Model</th>
+    <th colspan="2">Zero-shot Captioning</th>
+    <th colspan="5">General VQA</th>
+  </tr>
+  <tr>
+    <th>NoCaps</th>
+    <th>Flickr30K</th>
+    <th>VQAv2<sup>dev</sup></th>
+    <th>OK-VQA</th>
+    <th>GQA</th>
+    <th>SciQA-Img<br>(0-shot)</th>
+    <th>VizWiz<br>(0-shot)</th>
+  </tr>
+</thead>
+<tbody align="center">
+  <tr>
+    <td rowspan="10">Generalist<br>Models</td>
+    <td>Flamingo-9B</td>
+    <td>-</td>
+    <td>61.5</td>
+    <td>51.8</td>
+    <td>44.7</td>
+    <td>-</td>
+    <td>-</td>
+    <td>28.8</td>
+  </tr>
+  <tr>
+    <td>Flamingo-80B</td>
+    <td>-</td>
+    <td>67.2</td>
+    <td>56.3</td>
+    <td>50.6</td>
+    <td>-</td>
+    <td>-</td>
+    <td>31.6</td>
+  </tr>
+  <tr>
+    <td>Unified-IO-XL</td>
+    <td>100.0</td>
+    <td>-</td>
+    <td>77.9</td>
+    <td>54.0</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+  </tr>
+  <tr>
+    <td>Kosmos-1</td>
+    <td>-</td>
+    <td>67.1</td>
+    <td>51.0</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>29.2</td>
+  </tr>
+  <tr>
+    <td>Kosmos-2</td>
+    <td>-</td>
+    <td>66.7</td>
+    <td>45.6</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+  </tr>
+  <tr>
+    <td>BLIP-2 (Vicuna-13B)</td>
+    <td>103.9</td>
+    <td>71.6</td>
+    <td>65.0</td>
+    <td>45.9</td>
+    <td>32.3</td>
+    <td>61.0</td>
+    <td>19.6</td>
+  </tr>
+  <tr>
+    <td>InstructBLIP (Vicuna-13B)</td>
+    <td><strong>121.9</strong></td>
+    <td>82.8</td>
+    <td>-</td>
+    <td>-</td>
+    <td>49.5</td>
+    <td>63.1</td>
+    <td>33.4</td>
+  </tr>
+  <tr>
+    <td>Shikra (Vicuna-13B)</td>
+    <td>-</td>
+    <td>73.9</td>
+    <td>77.36</td>
+    <td>47.16</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+  </tr>
+  <tr>
+    <td><strong>Qwen-VL (Qwen-7B)</strong></td>
+    <td>121.4</td>
+    <td><b>85.8</b></td>
+    <td><b>78.8</b></td>
+    <td><b>58.6</b></td>
+    <td><b>59.3</b></td>
+    <td>67.1</td>
+    <td>35.2</td>
+  </tr>
+  <!-- <tr>
+    <td>Qwen-VL (4-shot)</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>63.6</td>
+    <td>-</td>
+    <td>-</td>
+    <td>39.1</td>
+  </tr> -->
+  <tr>
+    <td>Qwen-VL-Chat</td>
+    <td>120.2</td>
+    <td>81.0</td>
+    <td>78.2</td>
+    <td>56.6</td>
+    <td>57.5</td>
+    <td><b>68.2</b></td>
+    <td><b>38.9</b></td>
+  </tr>
+  <!-- <tr>
+    <td>Qwen-VL-Chat (4-shot)</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>60.6</td>
+    <td>-</td>
+    <td>-</td>
+    <td>44.45</td>
+  </tr> -->
+  <tr>
+    <td>Previous SOTA<br>(Per Task Fine-tuning)</td>
+    <td>-</td>
+    <td>127.0<br>(PALI-17B)</td>
+    <td>84.5<br>(InstructBLIP<br>-FlanT5-XL)</td>
+    <td>86.1<br>(PALI-X<br>-55B)</td>
+    <td>66.1<br>(PALI-X<br>-55B)</td>
+    <td>72.1<br>(CFR)</td>
+    <td>92.53<br>(LLaVa+<br>GPT-4)</td>
+    <td>70.9<br>(PALI-X<br>-55B)</td>
+  </tr>
+</tbody>
+</table>
+
+- 在 Zero-shot Caption 中，Qwen-VL 在 Flickr30K 数据集上取得了 **SOTA** 的结果，并在 Nocaps 数据集上取得了和 InstructBlip 可竞争的结果。
+- 在 General VQA 中，Qwen-VL 取得了 LVLM 模型同等量级和设定下 **SOTA** 的结果。
+- For zero-shot image captioning, Qwen-VL achieves the **SOTA** on Flickr30K and competitive results on Nocaps with InstructBlip.
+- For general VQA, Qwen-VL achieves the **SOTA** under the same generalist LVLM scale settings.
+
+### 文本导向的视觉问答 (Text-oriented VQA)
+
+<table>
+<thead>
+  <tr>
+    <th>Model type</th>
+    <th>Model</th>
+    <th>TextVQA</th>
+    <th>DocVQA</th>
+    <th>ChartQA</th>
+    <th>AI2D</th>
+    <th>OCR-VQA</th>
+  </tr>
+</thead>
+<tbody align="center">
+  <tr>
+    <td rowspan="5">Generalist Models</td>
+    <td>BLIP-2 (Vicuna-13B)</td>
+    <td>42.4</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+  </tr>
+  <tr>
+    <td>InstructBLIP (Vicuna-13B)</td>
+    <td>50.7</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+  </tr>
+  <tr>
+    <td>mPLUG-DocOwl (LLaMA-7B)</td>
+    <td>52.6</td>
+    <td>62.2</td>
+    <td>57.4</td>
+    <td>-</td>
+    <td>-</td>
+  </tr>
+  <tr>
+    <td>Pic2Struct-Large (1.3B)</td>
+    <td>-</td>
+    <td><b>76.6</b></td>
+    <td>58.6</td>
+    <td>42.1</td>
+    <td>71.3</td>
+  </tr>
+  <tr>
+    <td>Qwen-VL (Qwen-7B)</td>
+    <td><b>63.8</b></td>
+    <td>65.1</td>
+    <td><b>65.7</b></td>
+    <td><b>62.3</b></td>
+    <td><b>75.7</b></td>
+  </tr>
+  <tr>
+    <td>Specialist SOTAs<br>(Specialist/Finetuned)</td>
+    <td>PALI-X-55B (Single-task FT)<br>(Without OCR Pipeline)</td>
+    <td>71.44</td>
+    <td>80.0</td>
+    <td>70.0</td>
+    <td>81.2</td>
+    <td>75.0</td>
+  </tr>
+</tbody>
+</table>
+
+- 在文字相关的识别/问答评测上，取得了当前规模下通用 LVLM 达到的最好结果。
+- 分辨率对上述某几个评测非常重要，大部分 224 分辨率的开源 LVLM 模型无法完成以上评测，或只能通过切图的方式解决。Qwen-VL 将分辨率提升到 448，可以直接以端到端的方式进行以上评测。Qwen-VL 在很多任务上甚至超过了 1024 分辨率的 Pic2Struct-Large 模型。
+- In text-related recognition/QA evaluation, Qwen-VL achieves the SOTA under the generalist LVLM scale settings.
+- Resolution is important for several above evaluations. While most open-source LVLM models with 224 resolution are incapable of these evaluations or can only solve these by cutting images, Qwen-VL scales the resolution to 448 so that it can be evaluated end-to-end. Qwen-VL even outperforms Pic2Struct-Large models of 1024 resolution on some tasks.
+
+### 细粒度视觉定位 (Referring Expression Comprehension)
+
+<table>
+<thead>
+  <tr>
+    <th rowspan="2">Model type</th>
+    <th rowspan="2">Model</th>
+    <th colspan="3">RefCOCO</th>
+    <th colspan="3">RefCOCO+</th>
+    <th colspan="2">RefCOCOg</th>
+    <th>GRIT</th>
+  </tr>
+  <tr>
+    <th>val</th>
+    <th>test-A</th>
+    <th>test-B</th>
+    <th>val</th>
+    <th>test-A</th>
+    <th>test-B</th>
+    <th>val-u</th>
+    <th>test-u</th>
+    <th>refexp</th>
+  </tr>
+</thead>
+<tbody align="center">
+  <tr>
+    <td rowspan="8">Generalist Models</td>
+    <td>GPV-2</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>51.50</td>
+  </tr>
+  <tr>
+    <td>OFA-L*</td>
+    <td>79.96</td>
+    <td>83.67</td>
+    <td>76.39</td>
+    <td>68.29</td>
+    <td>76.00</td>
+    <td>61.75</td>
+    <td>67.57</td>
+    <td>67.58</td>
+    <td>61.70</td>
+  </tr>
+  <tr>
+    <td>Unified-IO</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td><b>78.61</b></td>
+  </tr>
+  <tr>
+    <td>VisionLLM-H</td>
+    <td></td>
+    <td>86.70</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+  </tr>
+  <tr>
+    <td>Shikra-7B</td>
+    <td>87.01</td>
+    <td>90.61</td>
+    <td>80.24 </td>
+    <td>81.60</td>
+    <td>87.36</td>
+    <td>72.12</td>
+    <td>82.27</td>
+    <td>82.19</td>
+    <td>69.34</td>
+  </tr>
+  <tr>
+    <td>Shikra-13B</td>
+    <td>87.83 </td>
+    <td>91.11</td>
+    <td>81.81</td>
+    <td>82.89</td>
+    <td>87.79</td>
+    <td>74.41</td>
+    <td>82.64</td>
+    <td>83.16</td>
+    <td>69.03</td>
+  </tr>
+  <tr>
+    <td>Qwen-VL-7B</td>
+    <td><b>89.36</b></td>
+    <td>92.26</td>
+    <td><b>85.34</b></td>
+    <td><b>83.12</b></td>
+    <td>88.25</td>
+    <td><b>77.21</b></td>
+    <td>85.58</td>
+    <td>85.48</td>
+    <td>78.22</td>
+  </tr>
+  <tr>
+    <td>Qwen-VL-7B-Chat</td>
+    <td>88.55</td>
+    <td><b>92.27</b></td>
+    <td>84.51</td>
+    <td>82.82</td>
+    <td><b>88.59</b></td>
+    <td>76.79</td>
+    <td><b>85.96</b></td>
+    <td><b>86.32</b></td>
+    <td>-</td>
+  <tr>
+    <td rowspan="3">Specialist SOTAs<br>(Specialist/Finetuned)</td>
+    <td>G-DINO-L</td>
+    <td>90.56&nbsp;&nbsp;</td>
+    <td>93.19</td>
+    <td>88.24</td>
+    <td>82.75</td>
+    <td>88.95</td>
+    <td>75.92</td>
+    <td>86.13</td>
+    <td>87.02</td>
+    <td>-</td>
+  </tr>
+  <tr>
+    <td>UNINEXT-H</td>
+    <td>92.64 </td>
+    <td>94.33</td>
+    <td>91.46</td>
+    <td>85.24</td>
+    <td>89.63</td>
+    <td>79.79</td>
+    <td>88.73</td>
+    <td>89.37</td>
+    <td>-</td>
+  </tr>
+  <tr>
+    <td>ONE-PEACE</td>
+    <td>92.58 </td>
+    <td>94.18</td>
+    <td>89.26</td>
+    <td>88.77</td>
+    <td>92.21</td>
+    <td>83.23</td>
+    <td>89.22</td>
+    <td>89.27</td>
+    <td>-</td>
+  </tr>
+</tbody>
+</table>
+
+- 在定位任务上，Qwen-VL 全面超过 Shikra-13B，取得了目前 Generalist LVLM 模型上在 Refcoco 上的 **SOTA**。
+- Qwen-VL 并没有在任何中文定位数据上训练过，但通过中文 Caption 数据和 英文 Grounding 数据的训练，可以 Zero-shot 泛化出中文 Grounding 能力。
+
+我们提供了以上**所有**评测脚本以供复现我们的实验结果。请阅读 [eval/EVALUATION.md](eval/EVALUATION.md) 了解更多信息。
+
+- Qwen-VL achieves the **SOTA** in all above referring expression comprehension benchmarks.
+- Qwen-VL has not been trained on any Chinese grounding data, but it can still generalize to the Chinese Grounding tasks in a zero-shot way by training Chinese Caption data and English Grounding data.
+
+We provide all of the above evaluation scripts for reproducing our experimental results. Please read [eval/EVALUATION.md](eval/EVALUATION.md) for more information.
+
+### 闲聊能力测评 (Chat Evaluation)
+
+TouchStone 是一个基于 GPT4 打分来评测 LVLM 模型的图文对话能力和人类对齐水平的基准。它涵盖了 300+张图片、800+道题目、27个类别，包括基础属性、人物地标、视觉推理、诗歌创作、故事写作、商品比较、图片解题等**尽可能广泛的类别**。关于 TouchStone 的详细介绍，请参考[touchstone/README_CN.md](touchstone/README_CN.md)了解更多信息。
+
+TouchStone is a benchmark based on scoring with GPT4 to evaluate the abilities of the LVLM model on text-image dialogue and alignment levels with humans. It covers a total of 300+ images, 800+ questions, and 27 categories, such as attribute-based Q&A, celebrity recognition, writing poetry, summarizing multiple images, product comparison, math problem solving, etc. Please read [touchstone/README_CN.md](touchstone/README.md) for more information.
+
+#### 英语 (English)
+
+| Model         | Score |
+|---------------|-------|
+| PandaGPT      | 488.5 |
+| MiniGPT4      | 531.7 |
+| InstructBLIP  | 552.4 |
+| LLaMA-AdapterV2 | 590.1 |
+| mPLUG-Owl     | 605.4 |
+| LLaVA         | 602.7 |
+| Qwen-VL-Chat   | 645.2 |
+
+#### 中文 (Chinese)
+
+| Model         | Score |
+|---------------|-------|
+| VisualGLM     | 247.1 |
+| Qwen-VL-Chat   | 401.2 |
+
+Qwen-VL-Chat 模型在中英文的对齐评测中均取得当前 LVLM 模型下的最��结果。
+
+Qwen-VL-Chat has achieved the best results in both Chinese and English alignment evaluation.
+<br>
+
+## 常见问题 (FAQ)
+
+如遇到问题，敬请查阅 [FAQ](https://github.com/QwenLM/Qwen-VL/blob/master/FAQ_zh.md)以及issue区，如仍无法解决再提交issue。
+
+If you meet problems, please refer to [FAQ](https://github.com/QwenLM/Qwen-VL/blob/master/FAQ.md) and the issues first to search a solution before you launch a new issue.
+<br>
+
+## 使用协议 (License Agreement)
+
+研究人员与开发者可使用Qwen-VL和Qwen-VL-Chat或进行二次开发。我们同样允许商业使用，具体细节请查看[LICENSE](https://github.com/QwenLM/Qwen-VL/blob/master/LICENSE)。如需商用，请填写[问卷](https://dashscope.console.aliyun.com/openModelApply/qianwen)申请。
+
+Researchers and developers are free to use the codes and model weights of both Qwen-VL and Qwen-VL-Chat. We also allow their commercial use. Check our license at [LICENSE](LICENSE) for more details.
+<br>
+
+## 引用 (Citation)
+
+如果你觉得我们的论文和代码对你的研究有帮助，请考虑:star: 和引用 :pencil: :)
+
+If you find our paper and code useful in your research, please consider giving a star :star: and citation :pencil: :)
+
+```BibTeX
+@article{Qwen-VL,
+  title={Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities},
+  author={Bai, Jinze and Bai, Shuai and Yang, Shusheng and Wang, Shijie and Tan, Sinan and Wang, Peng and Lin, Junyang and Zhou, Chang and Zhou, Jingren},
+  journal={arXiv preprint arXiv:2308.12966},
+  year={2023}
+}
+```
+<br>
+
+## 联系我们 (Contact Us)
+
+如果你想给我们的研发团队和产品团队留言，请通过邮件（[email protected]）联系我们。
+
+If you are interested to leave a message to either our research team or product team, feel free to send an email to [email protected].
+
+```
+
+```
+

Qwen-VL-Chat/SimSun.ttf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ca4da082cd970f0c8abaa79f213ddcbc475f7b5afabcb81b385998f9ebfbb53f
+size 10499104

Qwen-VL-Chat/config.json

@@ -0,0 +1,49 @@
+{
+  "_name_or_path": "./",
+  "architectures": [
+    "QWenLMHeadModel"
+  ],
+  "attn_dropout_prob": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_qwen.QWenConfig",
+    "AutoModelForCausalLM": "modeling_qwen.QWenLMHeadModel"
+  },
+  "bf16": false,
+  "emb_dropout_prob": 0.0,
+  "fp16": false,
+  "fp32": false,
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 22016,
+  "kv_channels": 128,
+  "layer_norm_epsilon": 1e-06,
+  "max_position_embeddings": 8192,
+  "model_type": "qwen",
+  "no_bias": true,
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "onnx_safe": null,
+  "rotary_emb_base": 10000,
+  "rotary_pct": 1.0,
+  "scale_attn_weights": true,
+  "seq_length": 2048,
+  "tie_word_embeddings": false,
+  "tokenizer_type": "QWenTokenizer",
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.31.0",
+  "use_cache": true,
+  "use_dynamic_ntk": true,
+  "use_flash_attn": false,
+  "use_logn_attn": true,
+  "visual": {
+    "heads": 16,
+    "image_size": 448,
+    "image_start_id": 151857,
+    "layers": 48,
+    "mlp_ratio": 4.9231,
+    "output_dim": 4096,
+    "patch_size": 14,
+    "width": 1664
+  },
+  "vocab_size": 151936
+}

Qwen-VL-Chat/configuration_qwen.py

@@ -0,0 +1,65 @@
+# Copyright (c) Alibaba Cloud.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+from transformers import PretrainedConfig
+
+
+class QWenConfig(PretrainedConfig):
+    model_type = "qwen"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    def __init__(
+        self,
+        vocab_size=151936,
+        hidden_size=4096,
+        num_hidden_layers=32,
+        num_attention_heads=32,
+        emb_dropout_prob=0.0,
+        attn_dropout_prob=0.0,
+        layer_norm_epsilon=1e-6,
+        initializer_range=0.02,
+        max_position_embeddings=8192,
+        scale_attn_weights=True,
+        use_cache=True,
+        bf16=False,
+        fp16=False,
+        fp32=False,
+        kv_channels=128,
+        rotary_pct=1.0,
+        rotary_emb_base=10000,
+        use_dynamic_ntk=True,
+        use_logn_attn=True,
+        use_flash_attn="auto",
+        intermediate_size=22016,
+        no_bias=True,
+        tie_word_embeddings=False,
+        **kwargs,
+    ):
+        self.vocab_size = vocab_size
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.emb_dropout_prob = emb_dropout_prob
+        self.attn_dropout_prob = attn_dropout_prob
+        self.layer_norm_epsilon = layer_norm_epsilon
+        self.initializer_range = initializer_range
+        self.scale_attn_weights = scale_attn_weights
+        self.use_cache = use_cache
+        self.max_position_embeddings = max_position_embeddings
+        self.bf16 = bf16
+        self.fp16 = fp16
+        self.fp32 = fp32
+        self.kv_channels = kv_channels
+        self.rotary_pct = rotary_pct
+        self.rotary_emb_base = rotary_emb_base
+        self.use_dynamic_ntk = use_dynamic_ntk
+        self.use_logn_attn = use_logn_attn
+        self.use_flash_attn = use_flash_attn
+        self.no_bias = no_bias
+        super().__init__(
+            tie_word_embeddings=tie_word_embeddings,
+            **kwargs
+        )

Qwen-VL-Chat/generation_config.json

@@ -0,0 +1,11 @@
+{
+  "chat_format": "chatml",
+  "do_sample": true,
+  "eos_token_id": 151643,
+  "max_new_tokens": 512,
+  "max_window_size": 6144,
+  "pad_token_id": 151643,
+  "top_k": 0,
+  "top_p": 0.3,
+  "transformers_version": "4.31.0"
+}

Qwen-VL-Chat/modeling_qwen.py

@@ -0,0 +1,1162 @@
+# Copyright (c) Alibaba Cloud.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+import importlib
+import math
+from typing import TYPE_CHECKING, Optional, Tuple, Union, Callable, List, Any, Generator
+
+import torch
+import torch.nn.functional as F
+import torch.utils.checkpoint
+from torch.cuda.amp import autocast
+
+from torch.nn import CrossEntropyLoss
+from transformers import PreTrainedTokenizer, GenerationConfig, StoppingCriteriaList
+from transformers.generation.logits_process import LogitsProcessorList
+
+if TYPE_CHECKING:
+    from transformers.generation.streamers import BaseStreamer
+from transformers.generation.utils import GenerateOutput
+from transformers.modeling_outputs import (
+    BaseModelOutputWithPast,
+    CausalLMOutputWithPast,
+)
+from transformers.modeling_utils import PreTrainedModel
+from transformers.utils import logging
+
+try:
+    from einops import rearrange
+except ImportError:
+    rearrange = None
+from torch import nn
+
+SUPPORT_CUDA = torch.cuda.is_available()
+SUPPORT_BF16 = SUPPORT_CUDA and torch.cuda.is_bf16_supported()
+SUPPORT_FP16 = SUPPORT_CUDA and torch.cuda.get_device_capability(0)[0] >= 7
+
+from .configuration_qwen import QWenConfig
+from .qwen_generation_utils import (
+    HistoryType,
+    make_context,
+    decode_tokens,
+    get_stop_words_ids,
+    StopWordsLogitsProcessor,
+)
+from .visual import VisionTransformer
+
+
+logger = logging.get_logger(__name__)
+
+_CHECKPOINT_FOR_DOC = "qwen"
+_CONFIG_FOR_DOC = "QWenConfig"
+
+QWen_PRETRAINED_MODEL_ARCHIVE_LIST = ["qwen-7b"]
+
+_ERROR_BAD_CHAT_FORMAT = """\
+We detect you are probably using the pretrained model (rather than chat model) for chatting, since the chat_format in generation_config is not "chatml".
+If you are directly using the model downloaded from Huggingface, please make sure you are using our "Qwen/Qwen-7B-Chat" Huggingface model (rather than "Qwen/Qwen-7B") when you call model.chat().
+我们检测到您可能在使用预训练模型（而非chat模型）进行多轮chat，因为您当前在generation_config指定的chat_format，并未设置为我们在对话中所支持的"chatml"格式。
+如果您在直接使用我们从Huggingface提供的模型，请确保您在调用model.chat()时，使用的是"Qwen/Qwen-7B-Chat"模型（而非"Qwen/Qwen-7B"预训练模型）。
+"""
+
+_SENTINEL = object()
+_ERROR_STREAM_IN_CHAT = """\
+Pass argument `stream` to model.chat() is buggy, deprecated, and marked for removal. Please use model.chat_stream(...) instead of model.chat(..., stream=True).
+向model.chat()传入参数stream的用法可能存在Bug，该用法已被废弃，将在未来被移除。请使用model.chat_stream(...)代替model.chat(..., stream=True)。
+"""
+
+apply_rotary_emb_func = None
+rms_norm = None
+
+
+# Copied from transformers.models.bart.modeling_bart._make_causal_mask
+def _make_causal_mask(
+    input_ids_shape: torch.Size, dtype: torch.dtype, device: torch.device, past_key_values_length: int = 0
+):
+    """
+    Make causal mask used for bi-directional self-attention.
+    """
+    bsz, tgt_len = input_ids_shape
+    mask = torch.full((tgt_len, tgt_len), torch.finfo(dtype).min, device=device)
+    mask_cond = torch.arange(mask.size(-1), device=device)
+    mask.masked_fill_(mask_cond < (mask_cond + 1).view(mask.size(-1), 1), 0)
+    mask = mask.to(dtype)
+
+    if past_key_values_length > 0:
+        mask = torch.cat([torch.zeros(tgt_len, past_key_values_length, dtype=dtype, device=device), mask], dim=-1)
+    return mask[None, None, :, :].expand(bsz, 1, tgt_len, tgt_len + past_key_values_length)
+
+
+# Copied from transformers.models.bart.modeling_bart._expand_mask
+def _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None):
+    """
+    Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.
+    """
+    bsz, src_len = mask.size()
+    tgt_len = tgt_len if tgt_len is not None else src_len
+
+    expanded_mask = mask[:, None, None, :].expand(bsz, 1, tgt_len, src_len).to(dtype)
+
+    inverted_mask = 1.0 - expanded_mask
+
+    return inverted_mask.masked_fill(inverted_mask.to(torch.bool), torch.finfo(dtype).min)
+
+
+class QWenAttention(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+
+        self.register_buffer("masked_bias", torch.tensor(-1e4), persistent=False)
+        self.seq_length = config.seq_length
+
+        self.hidden_size = config.hidden_size
+        self.split_size = config.hidden_size
+        self.num_heads = config.num_attention_heads
+        self.head_dim = self.hidden_size // self.num_heads
+
+        self.scale_attn_weights = True
+
+        self.projection_size = config.kv_channels * config.num_attention_heads
+
+        assert self.projection_size % config.num_attention_heads == 0
+        self.hidden_size_per_attention_head = (
+            self.projection_size // config.num_attention_heads
+        )
+
+        self.c_attn = nn.Linear(config.hidden_size, 3 * self.projection_size)
+
+        self.c_proj = nn.Linear(
+            config.hidden_size, self.projection_size, bias=not config.no_bias
+        )
+
+        self.is_fp32 = not (config.bf16 or config.fp16)
+        self.bf16 = config.bf16
+
+        self.use_dynamic_ntk = config.use_dynamic_ntk
+        self.use_logn_attn = config.use_logn_attn
+
+        logn_list = [
+            math.log(i, self.seq_length) if i > self.seq_length else 1
+            for i in range(1, 32768)
+        ]
+        self.logn_tensor = torch.tensor(logn_list)[None, :, None, None]
+
+        self.attn_dropout = nn.Dropout(config.attn_dropout_prob)
+
+    def _attn(self, query, key, value, registered_causal_mask, attention_mask=None, head_mask=None):
+        attn_weights = torch.matmul(query, key.transpose(-1, -2))
+
+        if self.scale_attn_weights:
+            attn_weights = attn_weights / torch.full(
+                [],
+                value.size(-1) ** 0.5,
+                dtype=attn_weights.dtype,
+                device=attn_weights.device,
+            )
+
+        query_length, key_length = query.size(-2), key.size(-2)
+        # causal_mask = self.bias[
+        #     :, :, key_length - query_length : key_length, :key_length
+        # ]
+        # mask_value = torch.finfo(attn_weights.dtype).min
+        # mask_value = torch.full([], mask_value, dtype=attn_weights.dtype).to(
+        #     attn_weights.device
+        # )
+        # attn_weights = torch.where(
+        #     causal_mask, attn_weights.to(attn_weights.dtype), mask_value
+        # )
+        attn_weights = attn_weights + attention_mask
+
+        attn_weights = nn.functional.softmax(attn_weights, dim=-1)
+
+        attn_weights = attn_weights.type(value.dtype)
+        attn_weights = self.attn_dropout(attn_weights)
+
+        if head_mask is not None:
+            attn_weights = attn_weights * head_mask
+
+        attn_output = torch.matmul(attn_weights, value)
+        attn_output = attn_output.transpose(1, 2)
+
+        return attn_output, attn_weights
+
+    def _upcast_and_reordered_attn(
+        self, query, key, value, registered_causal_mask, attention_mask=None, head_mask=None
+    ):
+        bsz, num_heads, q_seq_len, dk = query.size()
+        _, _, k_seq_len, _ = key.size()
+
+        attn_weights = torch.empty(
+            bsz * num_heads,
+            q_seq_len,
+            k_seq_len,
+            dtype=torch.float32,
+            device=query.device,
+        )
+
+        scale_factor = 1.0
+        if self.scale_attn_weights:
+            scale_factor /= float(value.size(-1)) ** 0.5
+
+        with autocast(enabled=False):
+            q, k = query.reshape(-1, q_seq_len, dk), key.transpose(-1, -2).reshape(
+                -1, dk, k_seq_len
+            )
+            attn_weights = torch.baddbmm(
+                attn_weights, q.float(), k.float(), beta=0, alpha=scale_factor
+            )
+            attn_weights = attn_weights.reshape(bsz, num_heads, q_seq_len, k_seq_len)
+
+        query_length, key_length = query.size(-2), key.size(-2)
+        causal_mask = registered_causal_mask[
+            :, :, key_length - query_length : key_length, :key_length
+        ]
+        mask_value = torch.finfo(attn_weights.dtype).min
+        mask_value = torch.tensor(mask_value, dtype=attn_weights.dtype).to(
+            attn_weights.device
+        )
+        attn_weights = torch.where(causal_mask, attn_weights, mask_value)
+
+        if attention_mask is not None:
+            attn_weights = attn_weights + attention_mask
+
+        attn_weights = nn.functional.softmax(attn_weights, dim=-1)
+
+        if attn_weights.dtype != torch.float32:
+            raise RuntimeError(
+                "Error with upcasting, attn_weights does not have dtype torch.float32"
+            )
+        attn_weights = attn_weights.type(value.dtype)
+        attn_weights = self.attn_dropout(attn_weights)
+
+        if head_mask is not None:
+            attn_weights = attn_weights * head_mask
+
+        attn_output = torch.matmul(attn_weights, value)
+
+        return attn_output, attn_weights
+
+    def _split_heads(self, tensor, num_heads, attn_head_size):
+        new_shape = tensor.size()[:-1] + (num_heads, attn_head_size)
+        tensor = tensor.view(new_shape)
+        return tensor
+
+    def _merge_heads(self, tensor, num_heads, attn_head_size):
+        tensor = tensor.contiguous()
+        new_shape = tensor.size()[:-2] + (num_heads * attn_head_size,)
+        return tensor.view(new_shape)
+
+    def forward(
+        self,
+        hidden_states: Optional[Tuple[torch.FloatTensor]],
+        rotary_pos_emb: Optional[List[torch.Tensor]] = None,
+        registered_causal_mask: Optional[torch.Tensor] = None,
+        layer_past: Optional[Tuple[torch.Tensor]] = None,
+        attention_mask: Optional[torch.FloatTensor] = None,
+        head_mask: Optional[torch.FloatTensor] = None,
+        encoder_hidden_states: Optional[torch.Tensor] = None,
+        encoder_attention_mask: Optional[torch.FloatTensor] = None,
+        output_attentions: Optional[bool] = False,
+        use_cache: Optional[bool] = False,
+    ):
+
+        mixed_x_layer = self.c_attn(hidden_states)
+
+        query, key, value = mixed_x_layer.split(self.split_size, dim=2)
+
+        query = self._split_heads(query, self.num_heads, self.head_dim)
+        key = self._split_heads(key, self.num_heads, self.head_dim)
+        value = self._split_heads(value, self.num_heads, self.head_dim)
+
+        if rotary_pos_emb is not None:
+            cur_len = query.shape[1]
+            rotary_pos_emb = [i[:, -cur_len:, :, :] for i in rotary_pos_emb]
+            rotary_pos_emb = (rotary_pos_emb,) * 2
+            q_pos_emb, k_pos_emb = rotary_pos_emb
+            # Slice the pos emb for current inference
+            query = apply_rotary_pos_emb(query, q_pos_emb)
+            key = apply_rotary_pos_emb(key, k_pos_emb)
+
+        if layer_past is not None:
+            past_key, past_value = layer_past[0], layer_past[1]
+            key = torch.cat((past_key, key), dim=1)
+            value = torch.cat((past_value, value), dim=1)
+
+        if use_cache:
+            present = (key, value)
+        else:
+            present = None
+
+        if self.use_logn_attn and not self.training:
+            if self.logn_tensor.device != query.device or self.logn_tensor.dtype != query.dtype:
+                self.logn_tensor = self.logn_tensor.to(query.device).type_as(query)
+            seq_start = key.size(1) - query.size(1)
+            seq_end = key.size(1)
+            logn_tensor = self.logn_tensor[:, seq_start:seq_end, :, :]
+            query = query * logn_tensor.expand_as(query)
+
+        query = query.permute(0, 2, 1, 3)
+        key = key.permute(0, 2, 1, 3)
+        value = value.permute(0, 2, 1, 3)
+        attn_output, attn_weight = self._attn(
+            query, key, value, registered_causal_mask, attention_mask, head_mask
+        )
+        context_layer = self._merge_heads(
+            attn_output, self.num_heads, self.head_dim
+        )
+
+        attn_output = self.c_proj(context_layer)
+
+        outputs = (attn_output, present)
+        if output_attentions:
+            outputs += (attn_weight,)
+
+        return outputs
+
+
+class QWenMLP(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.w1 = nn.Linear(
+            config.hidden_size, config.intermediate_size // 2, bias=not config.no_bias
+        )
+        self.w2 = nn.Linear(
+            config.hidden_size, config.intermediate_size // 2, bias=not config.no_bias
+        )
+        ff_dim_in = config.intermediate_size // 2
+        self.c_proj = nn.Linear(ff_dim_in, config.hidden_size, bias=not config.no_bias)
+
+    def forward(self, hidden_states):
+        a1 = self.w1(hidden_states)
+        a2 = self.w2(hidden_states)
+        intermediate_parallel = a1 * F.silu(a2)
+        output = self.c_proj(intermediate_parallel)
+        return output
+
+class QWenBlock(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        hidden_size = config.hidden_size
+        self.bf16 = config.bf16
+
+        self.ln_1 = RMSNorm(
+            hidden_size,
+            eps=config.layer_norm_epsilon,
+        )
+        self.attn = QWenAttention(config)
+        self.ln_2 = RMSNorm(
+            hidden_size,
+            eps=config.layer_norm_epsilon,
+        )
+
+        self.mlp = QWenMLP(config)
+
+    def forward(
+        self,
+        hidden_states: Optional[Tuple[torch.FloatTensor]],
+        rotary_pos_emb: Optional[List[torch.Tensor]] = None,
+        registered_causal_mask: Optional[torch.Tensor] = None,
+        layer_past: Optional[Tuple[torch.Tensor]] = None,
+        attention_mask: Optional[torch.FloatTensor] = None,
+        head_mask: Optional[torch.FloatTensor] = None,
+        encoder_hidden_states: Optional[torch.Tensor] = None,
+        encoder_attention_mask: Optional[torch.FloatTensor] = None,
+        use_cache: Optional[bool] = False,
+        output_attentions: Optional[bool] = False,
+    ):
+        layernorm_output = self.ln_1(hidden_states)
+
+        attn_outputs = self.attn(
+            layernorm_output,
+            rotary_pos_emb,
+            registered_causal_mask=registered_causal_mask,
+            layer_past=layer_past,
+            attention_mask=attention_mask,
+            head_mask=head_mask,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+        )
+        attn_output = attn_outputs[0]
+
+        outputs = attn_outputs[1:]
+
+        residual = hidden_states
+        layernorm_input = attn_output + residual
+
+        layernorm_output = self.ln_2(layernorm_input)
+
+        residual = layernorm_input
+        mlp_output = self.mlp(layernorm_output)
+        hidden_states = residual + mlp_output
+
+        if use_cache:
+            outputs = (hidden_states,) + outputs
+        else:
+            outputs = (hidden_states,) + outputs[1:]
+
+        return outputs
+
+
+class QWenPreTrainedModel(PreTrainedModel):
+    config_class = QWenConfig
+    base_model_prefix = "transformer"
+    is_parallelizable = False
+    supports_gradient_checkpointing = True
+    _no_split_modules = ["QWenBlock"]
+
+    def __init__(self, *inputs, **kwargs):
+        super().__init__(*inputs, **kwargs)
+
+    def _init_weights(self, module):
+        """Initialize the weights."""
+        if isinstance(module, nn.Linear):
+            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+            if module.bias is not None:
+                module.bias.data.zero_()
+        elif isinstance(module, nn.Embedding):
+            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+            if module.padding_idx is not None:
+                module.weight.data[module.padding_idx].zero_()
+        elif isinstance(module, RMSNorm):
+            module.weight.data.fill_(1.0)
+
+        for name, p in module.named_parameters():
+            if name == "c_proj.weight":
+                p.data.normal_(
+                    mean=0.0,
+                    std=(
+                        self.config.initializer_range
+                        / math.sqrt(2 * self.config.num_hidden_layers)
+                    ),
+                )
+
+    def _set_gradient_checkpointing(self, module, value=False):
+        if isinstance(module, QWenModel):
+            module.gradient_checkpointing = value
+
+
+class QWenModel(QWenPreTrainedModel):
+    _keys_to_ignore_on_load_missing = ["attn.masked_bias"]
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.vocab_size = config.vocab_size
+        self.num_hidden_layers = config.num_hidden_layers
+        self.embed_dim = config.hidden_size
+
+        self.gradient_checkpointing = False
+        self.use_dynamic_ntk = config.use_dynamic_ntk
+        self.seq_length = config.seq_length
+
+        self.wte = nn.Embedding(self.vocab_size, self.embed_dim)
+
+        self.drop = nn.Dropout(config.emb_dropout_prob)
+
+        if config.rotary_pct == 1.0:
+            self.rotary_ndims = None
+        else:
+            assert config.rotary_pct < 1
+            self.rotary_ndims = int(
+                config.kv_channels * config.rotary_pct
+            )
+        dim = (
+            self.rotary_ndims
+            if self.rotary_ndims is not None
+            else config.kv_channels
+        )
+        self.rotary_emb = RotaryEmbedding(dim, base=config.rotary_emb_base)
+
+        self.use_flash_attn = config.use_flash_attn
+        self.is_fp32 = not (config.bf16 or config.fp16)
+        self.registered_causal_mask = None
+        # if (
+        #     self.use_flash_attn
+        #     and flash_attn_unpadded_func is not None
+        #     and not self.is_fp32
+        # ):
+        #     self.registered_causal_mask = None
+        # else:
+        #     max_positions = config.max_position_embeddings
+        #     self.register_buffer(
+        #         "registered_causal_mask",
+        #         torch.tril(
+        #             torch.ones((max_positions, max_positions), dtype=torch.bool)
+        #         ).view(1, 1, max_positions, max_positions),
+        #         persistent=False,
+        #     )
+
+        self.h = nn.ModuleList(
+            [
+                QWenBlock(
+                    config
+                )
+                for i in range(config.num_hidden_layers)
+            ]
+        )
+        self.ln_f = RMSNorm(
+            self.embed_dim,
+            eps=config.layer_norm_epsilon,
+        )
+
+        self.visual = VisionTransformer(**config.visual)
+
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.wte
+
+    def set_input_embeddings(self, new_embeddings):
+        self.wte = new_embeddings
+    
+    # Copied from transformers.models.bart.modeling_bart.BartDecoder._prepare_decoder_attention_mask
+    def _prepare_decoder_attention_mask(self, attention_mask, input_shape, inputs_embeds, past_key_values_length):
+        # create causal mask
+        # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]
+        combined_attention_mask = None
+        if input_shape[-1] > 1:
+            combined_attention_mask = _make_causal_mask(
+                input_shape,
+                inputs_embeds.dtype,
+                device=inputs_embeds.device,
+                past_key_values_length=past_key_values_length,
+            )
+
+        if attention_mask is not None:
+            # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]
+            expanded_attn_mask = _expand_mask(attention_mask, inputs_embeds.dtype, tgt_len=input_shape[-1]).to(
+                inputs_embeds.device
+            )
+            combined_attention_mask = (
+                expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask + combined_attention_mask
+            )
+
+        return combined_attention_mask
+
+
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[Tuple[Tuple[torch.Tensor]]] = None,
+        attention_mask: Optional[torch.FloatTensor] = None,
+        token_type_ids: Optional[torch.LongTensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        head_mask: Optional[torch.FloatTensor] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        encoder_hidden_states: Optional[torch.Tensor] = None,
+        encoder_attention_mask: Optional[torch.FloatTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+    ):
+        if past_key_values is None and torch.any(input_ids == self.config.visual['image_start_id']):
+            bos_pos = torch.where(input_ids == self.config.visual['image_start_id'])
+            eos_pos = torch.where(input_ids == self.config.visual['image_start_id'] + 1)
+            assert (bos_pos[0] == eos_pos[0]).all()
+            img_pos = torch.stack((bos_pos[0], bos_pos[1], eos_pos[1]), dim=1)
+            images = []
+            for i, a, b in img_pos:
+                image = input_ids[i][a + 1 : b - 1].tolist()
+                image = image[ : image.index(self.config.visual['image_start_id'] + 2)]
+                images.append(bytes(image).decode('utf-8'))
+
+            images = self.visual.encode(images)
+            assert images.shape[0] == len(images)
+            fake_images = None
+        elif self.training:
+            fake_images=torch.zeros(1,3,224,224).to(
+                dtype=self.visual.conv1.weight.dtype, device=self.visual.conv1.weight.device)
+            images = self.visual(fake_images)
+        else:
+            fake_images = None
+            images = None
+
+        output_attentions = (
+            output_attentions
+            if output_attentions is not None
+            else self.config.output_attentions
+        )
+        output_hidden_states = (
+            output_hidden_states
+            if output_hidden_states is not None
+            else self.config.output_hidden_states
+        )
+        use_cache = use_cache if use_cache is not None else self.config.use_cache
+        return_dict = (
+            return_dict if return_dict is not None else self.config.use_return_dict
+        )
+
+        if input_ids is not None and inputs_embeds is not None:
+            raise ValueError(
+                "You cannot specify both input_ids and inputs_embeds at the same time"
+            )
+        elif input_ids is not None:
+            input_shape = input_ids.size()
+            input_ids = input_ids.view(-1, input_shape[-1])
+            batch_size = input_ids.shape[0]
+        elif inputs_embeds is not None:
+            input_shape = inputs_embeds.size()[:-1]
+            batch_size = inputs_embeds.shape[0]
+        else:
+            raise ValueError("You have to specify either input_ids or inputs_embeds")
+
+        device = input_ids.device if input_ids is not None else inputs_embeds.device
+
+        if token_type_ids is not None:
+            token_type_ids = token_type_ids.view(-1, input_shape[-1])
+        if position_ids is not None:
+            position_ids = position_ids.view(-1, input_shape[-1])
+
+        if past_key_values is None:
+            past_length = 0
+            past_key_values = tuple([None] * len(self.h))
+        else:
+            past_length = past_key_values[0][0].size(-2)
+
+        if position_ids is None:
+            position_ids = torch.arange(
+                past_length,
+                input_shape[-1] + past_length,
+                dtype=torch.long,
+                device=device,
+            )
+            position_ids = position_ids.unsqueeze(0).view(-1, input_shape[-1])
+
+        encoder_attention_mask = None
+        head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)
+
+        if inputs_embeds is None:
+            inputs_embeds = self.wte(input_ids)
+
+        if batch_size <= 0:
+            raise ValueError("batch_size has to be defined and > 0")
+        attention_mask = self._prepare_decoder_attention_mask(
+            attention_mask, input_shape, inputs_embeds, past_length
+        )
+
+        hidden_states = inputs_embeds
+
+        kv_seq_len = hidden_states.size()[1]
+        if past_key_values[0] is not None:
+            # past key values[0][0] shape: bs * seq_len * head_num * dim
+            kv_seq_len += past_key_values[0][0].shape[1]
+        if (
+            self.use_dynamic_ntk
+            and kv_seq_len == hidden_states.size()[1]
+            and not self.training
+        ):
+            context_value = math.log(kv_seq_len / self.seq_length, 2) + 1
+            ntk_alpha = 2 ** math.ceil(context_value) - 1
+            ntk_alpha = max(ntk_alpha, 1)
+        else:
+            ntk_alpha = self.rotary_emb._ntk_alpha_cached
+
+        rotary_pos_emb = self.rotary_emb(kv_seq_len, ntk_alpha=ntk_alpha)
+        for idx in range(len(rotary_pos_emb)):
+            rotary_pos_emb[idx] = rotary_pos_emb[idx].to(hidden_states.device)
+
+        hidden_states = self.drop(hidden_states).clone()
+        if fake_images is not None:
+            hidden_states = hidden_states + images.mean()*0
+        elif images is not None:
+            for idx, (i, a, b) in enumerate(img_pos):
+                hidden_states[i][a + 1 : b] = images[idx]
+        output_shape = input_shape + (hidden_states.size(-1),)
+
+        if self.gradient_checkpointing and self.training:
+            if use_cache:
+                logger.warning_once(
+                    "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
+                )
+                use_cache = False
+
+        presents = () if use_cache else None
+        all_self_attentions = () if output_attentions else None
+        all_hidden_states = () if output_hidden_states else None
+        for i, (block, layer_past) in enumerate(zip(self.h, past_key_values)):
+
+            if output_hidden_states:
+                all_hidden_states = all_hidden_states + (hidden_states,)
+
+            if self.gradient_checkpointing and self.training:
+
+                def create_custom_forward(module):
+                    def custom_forward(*inputs):
+                        # None for past_key_value
+                        return module(*inputs, use_cache, output_attentions)
+
+                    return custom_forward
+
+                outputs = torch.utils.checkpoint.checkpoint(
+                    create_custom_forward(block),
+                    hidden_states,
+                    rotary_pos_emb,
+                    self.registered_causal_mask,
+                    None,
+                    attention_mask,
+                    head_mask[i],
+                    encoder_hidden_states,
+                    encoder_attention_mask,
+                )
+            else:
+                outputs = block(
+                    hidden_states,
+                    layer_past=layer_past,
+                    rotary_pos_emb=rotary_pos_emb,
+                    registered_causal_mask=self.registered_causal_mask,
+                    attention_mask=attention_mask,
+                    head_mask=head_mask[i],
+                    encoder_hidden_states=encoder_hidden_states,
+                    encoder_attention_mask=encoder_attention_mask,
+                    use_cache=use_cache,
+                    output_attentions=output_attentions,
+                )
+
+            hidden_states = outputs[0]
+            if use_cache is True:
+                presents = presents + (outputs[1],)
+
+            if output_attentions:
+                all_self_attentions = all_self_attentions + (outputs[2 if use_cache else 1],)
+
+        hidden_states = self.ln_f(hidden_states)
+        hidden_states = hidden_states.view(output_shape)
+        # Add last hidden state
+        if output_hidden_states:
+            all_hidden_states = all_hidden_states + (hidden_states,)
+
+        if not return_dict:
+            return tuple(
+                v for v in [hidden_states, presents, all_hidden_states] if v is not None
+            )
+
+        return BaseModelOutputWithPast(
+            last_hidden_state=hidden_states,
+            past_key_values=presents,
+            hidden_states=all_hidden_states,
+            attentions=all_self_attentions,
+        )
+
+
+class QWenLMHeadModel(QWenPreTrainedModel):
+    _keys_to_ignore_on_load_missing = [r"h\.\d+\.attn\.rotary_emb\.inv_freq"]
+    _keys_to_ignore_on_load_unexpected = [r"h\.\d+\.attn\.masked_bias"]
+
+    def __init__(self, config):
+        super().__init__(config)
+        assert (
+            config.bf16 + config.fp16 + config.fp32 <= 1
+        ), "Only one of \"bf16\", \"fp16\", \"fp32\" can be true"
+
+        autoset_precision = config.bf16 + config.fp16 + config.fp32 == 0
+
+        if autoset_precision:
+            if SUPPORT_BF16:
+                logger.warn(
+                    "The model is automatically converting to bf16 for faster inference. "
+                    "If you want to disable the automatic precision, please manually add bf16/fp16/fp32=True to \"AutoModelForCausalLM.from_pretrained\"."
+                )
+                config.bf16 = True
+            elif SUPPORT_FP16:
+                logger.warn(
+                    "The model is automatically converting to fp16 for faster inference. "
+                    "If you want to disable the automatic precision, please manually add bf16/fp16/fp32=True to \"AutoModelForCausalLM.from_pretrained\"."
+                )
+                config.fp16 = True
+            else:
+                config.fp32 = True
+
+        if config.bf16 and SUPPORT_CUDA and not SUPPORT_BF16:
+            logger.warn("Your device does NOT seem to support bf16, you can switch to fp16 or fp32 by by passing fp16/fp32=True in \"AutoModelForCausalLM.from_pretrained\".")
+        if config.fp16 and SUPPORT_CUDA and not SUPPORT_FP16:
+            logger.warn("Your device does NOT support faster inference with fp16, please switch to fp32 which is likely to be faster")
+        if config.fp32:
+            if SUPPORT_BF16:
+                logger.warn("Your device support faster inference by passing bf16=True in \"AutoModelForCausalLM.from_pretrained\".")
+            elif SUPPORT_FP16:
+                logger.warn("Your device support faster inference by passing fp16=True in \"AutoModelForCausalLM.from_pretrained\".")
+
+        self.transformer = QWenModel(config)
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+
+        if config.bf16:
+            self.transformer.bfloat16()
+            self.lm_head.bfloat16()
+        if config.fp16:
+            self.transformer.half()
+            self.lm_head.half()
+        self.post_init()
+
+    def get_output_embeddings(self):
+        return self.lm_head
+
+    def set_output_embeddings(self, new_embeddings):
+        self.lm_head = new_embeddings
+
+    def prepare_inputs_for_generation(
+        self, input_ids, past_key_values=None, inputs_embeds=None, **kwargs
+    ):
+        token_type_ids = kwargs.get("token_type_ids", None)
+        if past_key_values:
+            input_ids = input_ids[:, -1].unsqueeze(-1)
+            if token_type_ids is not None:
+                token_type_ids = token_type_ids[:, -1].unsqueeze(-1)
+
+        attention_mask = kwargs.get("attention_mask", None)
+        position_ids = kwargs.get("position_ids", None)
+
+        if attention_mask is not None and position_ids is None:
+            position_ids = attention_mask.long().cumsum(-1) - 1
+            position_ids.masked_fill_(attention_mask == 0, 1)
+            if past_key_values:
+                position_ids = position_ids[:, -1].unsqueeze(-1)
+        else:
+            position_ids = None
+
+        if inputs_embeds is not None and past_key_values is None:
+            model_inputs = {"inputs_embeds": inputs_embeds}
+        else:
+            model_inputs = {"input_ids": input_ids}
+
+        model_inputs.update(
+            {
+                "past_key_values": past_key_values,
+                "use_cache": kwargs.get("use_cache"),
+                "position_ids": position_ids,
+                "attention_mask": attention_mask,
+                "token_type_ids": token_type_ids,
+            }
+        )
+        return model_inputs
+
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[Tuple[Tuple[torch.Tensor]]] = None,
+        attention_mask: Optional[torch.FloatTensor] = None,
+        token_type_ids: Optional[torch.LongTensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        head_mask: Optional[torch.FloatTensor] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        encoder_hidden_states: Optional[torch.Tensor] = None,
+        encoder_attention_mask: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+    ) -> Union[Tuple, CausalLMOutputWithPast]:
+
+        return_dict = (
+            return_dict if return_dict is not None else self.config.use_return_dict
+        )
+
+        transformer_outputs = self.transformer(
+            input_ids,
+            past_key_values=past_key_values,
+            attention_mask=attention_mask,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            head_mask=head_mask,
+            inputs_embeds=inputs_embeds,
+            encoder_hidden_states=encoder_hidden_states,
+            encoder_attention_mask=encoder_attention_mask,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+        hidden_states = transformer_outputs[0]
+
+        lm_logits = self.lm_head(hidden_states)
+
+        loss = None
+        if labels is not None:
+            labels = labels.to(lm_logits.device)
+            shift_logits = lm_logits[..., :-1, :].contiguous()
+            shift_labels = labels[..., 1:].contiguous()
+            loss_fct = CrossEntropyLoss()
+            loss = loss_fct(
+                shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1)
+            )
+
+        if not return_dict:
+            output = (lm_logits,) + transformer_outputs[1:]
+            return ((loss,) + output) if loss is not None else output
+
+        return CausalLMOutputWithPast(
+            loss=loss,
+            logits=lm_logits,
+            past_key_values=transformer_outputs.past_key_values,
+            hidden_states=transformer_outputs.hidden_states,
+            attentions=transformer_outputs.attentions,
+        )
+
+    @staticmethod
+    def _reorder_cache(
+        past_key_values: Tuple[Tuple[torch.Tensor]], beam_idx: torch.Tensor
+    ) -> Tuple[Tuple[torch.Tensor]]:
+
+        return tuple(
+            tuple(
+                past_state.index_select(0, beam_idx.to(past_state.device))
+                for past_state in layer_past
+            )
+            for layer_past in past_key_values
+        )
+
+    def chat(
+        self,
+        tokenizer: PreTrainedTokenizer,
+        query: str,
+        history: Optional[HistoryType],
+        system: str = "You are a helpful assistant.",
+        append_history: bool = True,
+        stream: Optional[bool] = _SENTINEL,
+        stop_words_ids: Optional[List[List[int]]] = None,
+        generation_config: Optional[GenerationConfig] = None,
+        **kwargs,
+    ) -> Tuple[str, HistoryType]:
+        generation_config = generation_config if generation_config is not None else self.generation_config
+
+        assert stream is _SENTINEL, _ERROR_STREAM_IN_CHAT
+        assert generation_config.chat_format == 'chatml', _ERROR_BAD_CHAT_FORMAT
+        if history is None:
+            history = []
+        if stop_words_ids is None:
+            stop_words_ids = []
+
+        max_window_size = kwargs.get('max_window_size', None)
+        if max_window_size is None:
+            max_window_size = generation_config.max_window_size
+        raw_text, context_tokens = make_context(
+            tokenizer,
+            query,
+            history=history,
+            system=system,
+            max_window_size=max_window_size,
+            chat_format=generation_config.chat_format,
+        )
+
+        stop_words_ids.extend(get_stop_words_ids(
+            generation_config.chat_format, tokenizer
+        ))
+        input_ids = torch.tensor([context_tokens]).to(self.device)
+        outputs = self.generate(
+                    input_ids,
+                    stop_words_ids=stop_words_ids,
+                    return_dict_in_generate=False,
+                    generation_config=generation_config,
+                    **kwargs,
+                )
+
+        response = decode_tokens(
+            outputs[0],
+            tokenizer,
+            raw_text_len=len(raw_text),
+            context_length=len(context_tokens),
+            chat_format=generation_config.chat_format,
+            verbose=False,
+            errors='replace'
+        )
+
+        if append_history:
+            history.append((query, response))
+
+        return response, history
+
+    def chat_stream(
+            self,
+            tokenizer: PreTrainedTokenizer,
+            query: str,
+            history: Optional[HistoryType],
+            system: str = "You are a helpful assistant.",
+            stop_words_ids: Optional[List[List[int]]] = None,
+            logits_processor: Optional[LogitsProcessorList] = None,
+            generation_config: Optional[GenerationConfig] = None,
+            **kwargs,
+    ) -> Generator[str, Any, None]:
+        generation_config = generation_config if generation_config is not None else self.generation_config
+        assert generation_config.chat_format == 'chatml', _ERROR_BAD_CHAT_FORMAT
+        if history is None:
+            history = []
+        if stop_words_ids is None:
+            stop_words_ids = []
+
+        max_window_size = kwargs.get('max_window_size', None)
+        if max_window_size is None:
+            max_window_size = generation_config.max_window_size
+        raw_text, context_tokens = make_context(
+            tokenizer,
+            query,
+            history=history,
+            system=system,
+            max_window_size=max_window_size,
+            chat_format=generation_config.chat_format,
+        )
+
+        stop_words_ids.extend(get_stop_words_ids(
+            generation_config.chat_format, tokenizer
+        ))
+        if stop_words_ids is not None:
+            stop_words_logits_processor = StopWordsLogitsProcessor(
+                stop_words_ids=stop_words_ids,
+                eos_token_id=generation_config.eos_token_id,
+            )
+            if logits_processor is None:
+                logits_processor = LogitsProcessorList([stop_words_logits_processor])
+            else:
+                logits_processor.append(stop_words_logits_processor)
+        input_ids = torch.tensor([context_tokens]).to(self.device)
+
+        from transformers_stream_generator.main import NewGenerationMixin, StreamGenerationConfig
+        self.__class__.generate_stream = NewGenerationMixin.generate
+        self.__class__.sample_stream = NewGenerationMixin.sample_stream
+        stream_config = StreamGenerationConfig(**generation_config.to_dict(), do_stream=True)
+
+        def stream_generator():
+            outputs = []
+            for token in self.generate_stream(
+                    input_ids,
+                    return_dict_in_generate=False,
+                    generation_config=stream_config,
+                    logits_processor=logits_processor,
+                    seed=-1,
+                    **kwargs):
+                outputs.append(token.item())
+                yield tokenizer.decode(outputs, skip_special_tokens=True, errors='ignore', keep_image_special=True)
+
+        return stream_generator()
+
+    def generate(
+        self,
+        inputs: Optional[torch.Tensor] = None,
+        generation_config: Optional[GenerationConfig] = None,
+        logits_processor: Optional[LogitsProcessorList] = None,
+        stopping_criteria: Optional[StoppingCriteriaList] = None,
+        prefix_allowed_tokens_fn: Optional[
+            Callable[[int, torch.Tensor], List[int]]
+        ] = None,
+        synced_gpus: Optional[bool] = None,
+        assistant_model: Optional["PreTrainedModel"] = None,
+        streamer: Optional["BaseStreamer"] = None,
+        **kwargs,
+    ) -> Union[GenerateOutput, torch.LongTensor]:
+        generation_config = generation_config if generation_config is not None else self.generation_config
+
+        # Process stop_words_ids.
+        stop_words_ids = kwargs.pop("stop_words_ids", None)
+        if stop_words_ids is None and generation_config is not None:
+            stop_words_ids = getattr(generation_config, "stop_words_ids", None)
+        if stop_words_ids is None:
+            stop_words_ids = getattr(generation_config, "stop_words_ids", None)
+
+        if stop_words_ids is not None:
+            stop_words_logits_processor = StopWordsLogitsProcessor(
+                stop_words_ids=stop_words_ids,
+                eos_token_id=generation_config.eos_token_id,
+            )
+            if logits_processor is None:
+                logits_processor = LogitsProcessorList([stop_words_logits_processor])
+            else:
+                logits_processor.append(stop_words_logits_processor)
+
+        return super().generate(
+            inputs,
+            generation_config=generation_config,
+            logits_processor=logits_processor,
+            stopping_criteria=stopping_criteria,
+            prefix_allowed_tokens_fn=prefix_allowed_tokens_fn,
+            synced_gpus=synced_gpus,
+            assistant_model=assistant_model,
+            streamer=streamer,
+            **kwargs,
+        )
+
+
+class RotaryEmbedding(torch.nn.Module):
+    def __init__(self, dim, base=10000):
+        super().__init__()
+        self.dim = dim
+        self.base = base
+        self.inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2).float() / dim))
+        if importlib.util.find_spec("einops") is None:
+            raise RuntimeError("einops is required for Rotary Embedding")
+
+        self._rotary_pos_emb_cache = None
+        self._seq_len_cached = 0
+        self._ntk_alpha_cached = 1.0
+
+    def update_rotary_pos_emb_cache(self, max_seq_len, offset=0, ntk_alpha=1.0):
+        seqlen = max_seq_len + offset
+        if seqlen > self._seq_len_cached or ntk_alpha != self._ntk_alpha_cached:
+            base = self.base * ntk_alpha ** (self.dim / (self.dim - 2))
+            self.inv_freq = 1.0 / (
+                base
+                ** (
+                    torch.arange(0, self.dim, 2, device=self.inv_freq.device).float()
+                    / self.dim
+                )
+            )
+            self._seq_len_cached = max(2 * seqlen, 16)
+            self._ntk_alpha_cached = ntk_alpha
+            seq = torch.arange(self._seq_len_cached, device=self.inv_freq.device)
+            freqs = torch.outer(seq.type_as(self.inv_freq), self.inv_freq)
+            
+            emb = torch.cat((freqs, freqs), dim=-1)
+            from einops import rearrange
+
+            emb = rearrange(emb, "n d -> 1 n 1 d")
+
+            cos, sin = emb.cos(), emb.sin()
+            self._rotary_pos_emb_cache = [cos, sin]
+
+    def forward(self, max_seq_len, offset=0, ntk_alpha=1.0):
+        self.update_rotary_pos_emb_cache(max_seq_len, offset, ntk_alpha)
+        cos, sin = self._rotary_pos_emb_cache
+        return [cos[:, offset : offset + max_seq_len], sin[:, offset : offset + max_seq_len]]
+
+
+def _rotate_half(x):
+    from einops import rearrange
+
+    x = rearrange(x, "... (j d) -> ... j d", j=2)
+    x1, x2 = x.unbind(dim=-2)
+    return torch.cat((-x2, x1), dim=-1)
+
+
+def apply_rotary_pos_emb(t, freqs):
+    cos, sin = freqs
+    if apply_rotary_emb_func is not None and t.is_cuda:
+        t_ = t.float()
+        cos = cos.squeeze(0).squeeze(1)[:, : cos.shape[-1] // 2]
+        sin = sin.squeeze(0).squeeze(1)[:, : sin.shape[-1] // 2]
+        output = apply_rotary_emb_func(t_, cos, sin).type_as(t)
+        return output
+    else:
+        rot_dim = freqs[0].shape[-1]
+        cos, sin = freqs
+        t_, t_pass_ = t[..., :rot_dim], t[..., rot_dim:]
+        t_ = t_.float()
+        t_pass_ = t_pass_.float()
+        t_ = (t_ * cos) + (_rotate_half(t_) * sin)
+        return torch.cat((t_, t_pass_), dim=-1).type_as(t)
+
+
+class RMSNorm(torch.nn.Module):
+    def __init__(self, dim: int, eps: float = 1e-6):
+        super().__init__()
+        self.eps = eps
+        self.weight = nn.Parameter(torch.ones(dim))
+
+    def _norm(self, x):
+        return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
+
+    def forward(self, x):
+        if rms_norm is not None and x.is_cuda:
+            return rms_norm(x, self.weight, self.eps)
+        else:
+            output = self._norm(x.float()).type_as(x)
+            return output * self.weight

Qwen-VL-Chat/pytorch_model-00001-of-00010.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d63e4b4238be3897d3b44d0f604422fc07dfceaf971ebde7adadd7be7a2a35bb
+size 1964070255

Qwen-VL-Chat/pytorch_model-00002-of-00010.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a14062dc6abcccda58a4d4ab346b0990a7fc6066b571eddfd96b92b0024c32be
+size 1933791629

Qwen-VL-Chat/pytorch_model-00003-of-00010.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:72051aea446b6961d937ff9586adb5aea2ba435d490daa31d1332bd6c7be682f
+size 1933791629

Qwen-VL-Chat/pytorch_model-00004-of-00010.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:01292b3d0c48b2141e247625699f5e5889655ffc4c7f2bcf92f3692c24b21234
+size 1990406267

Qwen-VL-Chat/pytorch_model-00005-of-00010.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:75544b895cfb4ac8f59c5bf27a17c355a6e380ec0b83ee27298cc2463abb9914
+size 1923281019

Qwen-VL-Chat/pytorch_model-00006-of-00010.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f31964509ef34c82ab95eaac98121eeac193a1828a53abc2801b75b9fced230
+size 1933783163

Qwen-VL-Chat/pytorch_model-00007-of-00010.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1dc41b294fe96db7800587308d9235970490e5862d83ee50407be61085589e24
+size 1933791693

Qwen-VL-Chat/pytorch_model-00008-of-00010.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:164383ddd886d8e18de072be7632c258e6127a885e73cc761056ade7ff62abca
+size 1975364797

Qwen-VL-Chat/pytorch_model-00009-of-00010.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:930b17b36aa050b7ed1853d65f8f2e72a9e944916fdb3f002a7f6d6963ec91be
+size 1994920659

Qwen-VL-Chat/pytorch_model-00010-of-00010.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1f6ecfbabe9f0868fa2fca4cf726d5d956850e516a12a95aea63b37abc2c27d6
+size 1730967567

Qwen-VL-Chat/pytorch_model.bin.index.json

@@ -0,0 +1,860 @@
+{
+  "metadata": {
+    "total_size": 19313870336
+  },
+  "weight_map": {
+    "lm_head.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.h.0.attn.c_attn.bias": "pytorch_model-00001-of-00010.bin",
+    "transformer.h.0.attn.c_attn.weight": "pytorch_model-00001-of-00010.bin",
+    "transformer.h.0.attn.c_proj.weight": "pytorch_model-00001-of-00010.bin",
+    "transformer.h.0.ln_1.weight": "pytorch_model-00001-of-00010.bin",
+    "transformer.h.0.ln_2.weight": "pytorch_model-00001-of-00010.bin",
+    "transformer.h.0.mlp.c_proj.weight": "pytorch_model-00001-of-00010.bin",
+    "transformer.h.0.mlp.w1.weight": "pytorch_model-00001-of-00010.bin",
+    "transformer.h.0.mlp.w2.weight": "pytorch_model-00001-of-00010.bin",
+    "transformer.h.1.attn.c_attn.bias": "pytorch_model-00001-of-00010.bin",
+    "transformer.h.1.attn.c_attn.weight": "pytorch_model-00001-of-00010.bin",
+    "transformer.h.1.attn.c_proj.weight": "pytorch_model-00001-of-00010.bin",
+    "transformer.h.1.ln_1.weight": "pytorch_model-00001-of-00010.bin",
+    "transformer.h.1.ln_2.weight": "pytorch_model-00001-of-00010.bin",
+    "transformer.h.1.mlp.c_proj.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.1.mlp.w1.weight": "pytorch_model-00001-of-00010.bin",
+    "transformer.h.1.mlp.w2.weight": "pytorch_model-00001-of-00010.bin",
+    "transformer.h.10.attn.c_attn.bias": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.10.attn.c_attn.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.10.attn.c_proj.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.10.ln_1.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.10.ln_2.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.10.mlp.c_proj.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.10.mlp.w1.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.10.mlp.w2.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.11.attn.c_attn.bias": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.11.attn.c_attn.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.11.attn.c_proj.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.11.ln_1.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.11.ln_2.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.11.mlp.c_proj.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.11.mlp.w1.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.11.mlp.w2.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.12.attn.c_attn.bias": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.12.attn.c_attn.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.12.attn.c_proj.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.12.ln_1.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.12.ln_2.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.12.mlp.c_proj.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.12.mlp.w1.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.12.mlp.w2.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.13.attn.c_attn.bias": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.13.attn.c_attn.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.13.attn.c_proj.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.13.ln_1.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.13.ln_2.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.13.mlp.c_proj.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.13.mlp.w1.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.13.mlp.w2.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.14.attn.c_attn.bias": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.14.attn.c_attn.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.14.attn.c_proj.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.14.ln_1.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.14.ln_2.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.14.mlp.c_proj.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.14.mlp.w1.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.14.mlp.w2.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.15.attn.c_attn.bias": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.15.attn.c_attn.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.15.attn.c_proj.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.15.ln_1.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.15.ln_2.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.15.mlp.c_proj.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.15.mlp.w1.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.15.mlp.w2.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.16.attn.c_attn.bias": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.16.attn.c_attn.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.16.attn.c_proj.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.16.ln_1.weight": "pytorch_model-00004-of-00010.bin",
+    "transformer.h.16.ln_2.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.16.mlp.c_proj.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.16.mlp.w1.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.16.mlp.w2.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.17.attn.c_attn.bias": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.17.attn.c_attn.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.17.attn.c_proj.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.17.ln_1.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.17.ln_2.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.17.mlp.c_proj.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.17.mlp.w1.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.17.mlp.w2.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.18.attn.c_attn.bias": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.18.attn.c_attn.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.18.attn.c_proj.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.18.ln_1.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.18.ln_2.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.18.mlp.c_proj.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.18.mlp.w1.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.18.mlp.w2.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.19.attn.c_attn.bias": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.19.attn.c_attn.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.19.attn.c_proj.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.19.ln_1.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.19.ln_2.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.19.mlp.c_proj.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.19.mlp.w1.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.19.mlp.w2.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.2.attn.c_attn.bias": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.2.attn.c_attn.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.2.attn.c_proj.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.2.ln_1.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.2.ln_2.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.2.mlp.c_proj.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.2.mlp.w1.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.2.mlp.w2.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.20.attn.c_attn.bias": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.20.attn.c_attn.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.20.attn.c_proj.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.20.ln_1.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.20.ln_2.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.20.mlp.c_proj.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.20.mlp.w1.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.20.mlp.w2.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.21.attn.c_attn.bias": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.21.attn.c_attn.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.21.attn.c_proj.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.21.ln_1.weight": "pytorch_model-00005-of-00010.bin",
+    "transformer.h.21.ln_2.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.21.mlp.c_proj.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.21.mlp.w1.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.21.mlp.w2.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.22.attn.c_attn.bias": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.22.attn.c_attn.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.22.attn.c_proj.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.22.ln_1.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.22.ln_2.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.22.mlp.c_proj.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.22.mlp.w1.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.22.mlp.w2.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.23.attn.c_attn.bias": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.23.attn.c_attn.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.23.attn.c_proj.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.23.ln_1.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.23.ln_2.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.23.mlp.c_proj.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.23.mlp.w1.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.23.mlp.w2.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.24.attn.c_attn.bias": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.24.attn.c_attn.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.24.attn.c_proj.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.24.ln_1.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.24.ln_2.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.24.mlp.c_proj.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.24.mlp.w1.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.24.mlp.w2.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.25.attn.c_attn.bias": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.25.attn.c_attn.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.25.attn.c_proj.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.25.ln_1.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.25.ln_2.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.25.mlp.c_proj.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.25.mlp.w1.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.25.mlp.w2.weight": "pytorch_model-00006-of-00010.bin",
+    "transformer.h.26.attn.c_attn.bias": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.26.attn.c_attn.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.26.attn.c_proj.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.26.ln_1.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.26.ln_2.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.26.mlp.c_proj.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.26.mlp.w1.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.26.mlp.w2.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.27.attn.c_attn.bias": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.27.attn.c_attn.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.27.attn.c_proj.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.27.ln_1.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.27.ln_2.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.27.mlp.c_proj.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.27.mlp.w1.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.27.mlp.w2.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.28.attn.c_attn.bias": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.28.attn.c_attn.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.28.attn.c_proj.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.28.ln_1.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.28.ln_2.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.28.mlp.c_proj.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.28.mlp.w1.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.28.mlp.w2.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.29.attn.c_attn.bias": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.29.attn.c_attn.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.29.attn.c_proj.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.29.ln_1.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.29.ln_2.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.29.mlp.c_proj.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.29.mlp.w1.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.29.mlp.w2.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.3.attn.c_attn.bias": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.3.attn.c_attn.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.3.attn.c_proj.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.3.ln_1.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.3.ln_2.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.3.mlp.c_proj.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.3.mlp.w1.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.3.mlp.w2.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.30.attn.c_attn.bias": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.30.attn.c_attn.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.30.attn.c_proj.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.30.ln_1.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.30.ln_2.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.30.mlp.c_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.h.30.mlp.w1.weight": "pytorch_model-00007-of-00010.bin",
+    "transformer.h.30.mlp.w2.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.h.31.attn.c_attn.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.h.31.attn.c_attn.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.h.31.attn.c_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.h.31.ln_1.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.h.31.ln_2.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.h.31.mlp.c_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.h.31.mlp.w1.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.h.31.mlp.w2.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.h.4.attn.c_attn.bias": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.4.attn.c_attn.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.4.attn.c_proj.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.4.ln_1.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.4.ln_2.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.4.mlp.c_proj.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.4.mlp.w1.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.4.mlp.w2.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.5.attn.c_attn.bias": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.5.attn.c_attn.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.5.attn.c_proj.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.5.ln_1.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.5.ln_2.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.5.mlp.c_proj.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.5.mlp.w1.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.5.mlp.w2.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.6.attn.c_attn.bias": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.6.attn.c_attn.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.6.attn.c_proj.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.6.ln_1.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.6.ln_2.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.6.mlp.c_proj.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.6.mlp.w1.weight": "pytorch_model-00002-of-00010.bin",
+    "transformer.h.6.mlp.w2.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.7.attn.c_attn.bias": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.7.attn.c_attn.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.7.attn.c_proj.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.7.ln_1.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.7.ln_2.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.7.mlp.c_proj.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.7.mlp.w1.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.7.mlp.w2.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.8.attn.c_attn.bias": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.8.attn.c_attn.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.8.attn.c_proj.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.8.ln_1.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.8.ln_2.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.8.mlp.c_proj.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.8.mlp.w1.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.8.mlp.w2.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.9.attn.c_attn.bias": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.9.attn.c_attn.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.9.attn.c_proj.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.9.ln_1.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.9.ln_2.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.9.mlp.c_proj.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.9.mlp.w1.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.h.9.mlp.w2.weight": "pytorch_model-00003-of-00010.bin",
+    "transformer.ln_f.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.attn_pool.attn.in_proj_bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.attn_pool.attn.in_proj_weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.attn_pool.attn.out_proj.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.attn_pool.attn.out_proj.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.attn_pool.kv_proj.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.attn_pool.ln_kv.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.attn_pool.ln_kv.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.attn_pool.ln_q.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.attn_pool.ln_q.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.attn_pool.pos_embed": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.attn_pool.query": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.conv1.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.ln_post.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.ln_post.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.ln_pre.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.ln_pre.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.positional_embedding": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.proj": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.0.attn.in_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.0.attn.in_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.0.attn.out_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.0.attn.out_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.0.ln_1.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.0.ln_1.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.0.ln_2.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.0.ln_2.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.0.mlp.c_fc.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.0.mlp.c_fc.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.0.mlp.c_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.0.mlp.c_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.1.attn.in_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.1.attn.in_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.1.attn.out_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.1.attn.out_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.1.ln_1.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.1.ln_1.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.1.ln_2.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.1.ln_2.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.1.mlp.c_fc.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.1.mlp.c_fc.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.1.mlp.c_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.1.mlp.c_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.10.attn.in_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.10.attn.in_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.10.attn.out_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.10.attn.out_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.10.ln_1.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.10.ln_1.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.10.ln_2.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.10.ln_2.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.10.mlp.c_fc.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.10.mlp.c_fc.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.10.mlp.c_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.10.mlp.c_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.11.attn.in_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.11.attn.in_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.11.attn.out_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.11.attn.out_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.11.ln_1.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.11.ln_1.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.11.ln_2.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.11.ln_2.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.11.mlp.c_fc.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.11.mlp.c_fc.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.11.mlp.c_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.11.mlp.c_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.12.attn.in_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.12.attn.in_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.12.attn.out_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.12.attn.out_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.12.ln_1.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.12.ln_1.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.12.ln_2.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.12.ln_2.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.12.mlp.c_fc.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.12.mlp.c_fc.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.12.mlp.c_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.12.mlp.c_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.13.attn.in_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.13.attn.in_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.13.attn.out_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.13.attn.out_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.13.ln_1.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.13.ln_1.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.13.ln_2.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.13.ln_2.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.13.mlp.c_fc.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.13.mlp.c_fc.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.13.mlp.c_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.13.mlp.c_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.14.attn.in_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.14.attn.in_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.14.attn.out_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.14.attn.out_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.14.ln_1.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.14.ln_1.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.14.ln_2.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.14.ln_2.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.14.mlp.c_fc.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.14.mlp.c_fc.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.14.mlp.c_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.14.mlp.c_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.15.attn.in_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.15.attn.in_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.15.attn.out_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.15.attn.out_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.15.ln_1.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.15.ln_1.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.15.ln_2.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.15.ln_2.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.15.mlp.c_fc.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.15.mlp.c_fc.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.15.mlp.c_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.15.mlp.c_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.16.attn.in_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.16.attn.in_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.16.attn.out_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.16.attn.out_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.16.ln_1.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.16.ln_1.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.16.ln_2.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.16.ln_2.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.16.mlp.c_fc.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.16.mlp.c_fc.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.16.mlp.c_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.16.mlp.c_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.17.attn.in_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.17.attn.in_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.17.attn.out_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.17.attn.out_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.17.ln_1.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.17.ln_1.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.17.ln_2.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.17.ln_2.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.17.mlp.c_fc.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.17.mlp.c_fc.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.17.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.17.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.18.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.18.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.18.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.18.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.18.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.18.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.18.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.18.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.18.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.18.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.18.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.18.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.19.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.19.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.19.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.19.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.19.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.19.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.19.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.19.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.19.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.19.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.19.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.19.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.2.attn.in_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.2.attn.in_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.2.attn.out_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.2.attn.out_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.2.ln_1.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.2.ln_1.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.2.ln_2.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.2.ln_2.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.2.mlp.c_fc.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.2.mlp.c_fc.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.2.mlp.c_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.2.mlp.c_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.20.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.20.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.20.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.20.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.20.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.20.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.20.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.20.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.20.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.20.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.20.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.20.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.21.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.21.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.21.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.21.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.21.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.21.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.21.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.21.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.21.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.21.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.21.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.21.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.22.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.22.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.22.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.22.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.22.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.22.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.22.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.22.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.22.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.22.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.22.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.22.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.23.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.23.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.23.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.23.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.23.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.23.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.23.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.23.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.23.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.23.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.23.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.23.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.24.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.24.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.24.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.24.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.24.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.24.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.24.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.24.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.24.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.24.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.24.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.24.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.25.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.25.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.25.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.25.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.25.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.25.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.25.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.25.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.25.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.25.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.25.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.25.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.26.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.26.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.26.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.26.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.26.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.26.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.26.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.26.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.26.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.26.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.26.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.26.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.27.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.27.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.27.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.27.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.27.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.27.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.27.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.27.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.27.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.27.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.27.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.27.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.28.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.28.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.28.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.28.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.28.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.28.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.28.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.28.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.28.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.28.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.28.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.28.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.29.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.29.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.29.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.29.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.29.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.29.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.29.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.29.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.29.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.29.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.29.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.29.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.3.attn.in_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.3.attn.in_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.3.attn.out_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.3.attn.out_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.3.ln_1.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.3.ln_1.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.3.ln_2.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.3.ln_2.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.3.mlp.c_fc.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.3.mlp.c_fc.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.3.mlp.c_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.3.mlp.c_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.30.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.30.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.30.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.30.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.30.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.30.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.30.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.30.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.30.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.30.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.30.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.30.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.31.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.31.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.31.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.31.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.31.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.31.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.31.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.31.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.31.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.31.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.31.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.31.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.32.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.32.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.32.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.32.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.32.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.32.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.32.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.32.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.32.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.32.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.32.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.32.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.33.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.33.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.33.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.33.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.33.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.33.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.33.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.33.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.33.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.33.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.33.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.33.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.34.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.34.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.34.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.34.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.34.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.34.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.34.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.34.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.34.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.34.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.34.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.34.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.35.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.35.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.35.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.35.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.35.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.35.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.35.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.35.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.35.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.35.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.35.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.35.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.36.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.36.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.36.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.36.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.36.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.36.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.36.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.36.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.36.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.36.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.36.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.36.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.37.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.37.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.37.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.37.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.37.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.37.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.37.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.37.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.37.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.37.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.37.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.37.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.38.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.38.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.38.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.38.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.38.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.38.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.38.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.38.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.38.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.38.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.38.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.38.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.39.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.39.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.39.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.39.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.39.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.39.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.39.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.39.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.39.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.39.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.39.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.39.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.4.attn.in_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.4.attn.in_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.4.attn.out_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.4.attn.out_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.4.ln_1.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.4.ln_1.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.4.ln_2.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.4.ln_2.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.4.mlp.c_fc.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.4.mlp.c_fc.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.4.mlp.c_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.4.mlp.c_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.40.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.40.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.40.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.40.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.40.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.40.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.40.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.40.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.40.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.40.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.40.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.40.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.41.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.41.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.41.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.41.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.41.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.41.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.41.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.41.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.41.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.41.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.41.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.41.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.42.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.42.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.42.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.42.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.42.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.42.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.42.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.42.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.42.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.42.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.42.mlp.c_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.42.mlp.c_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.43.attn.in_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.43.attn.in_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.43.attn.out_proj.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.43.attn.out_proj.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.43.ln_1.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.43.ln_1.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.43.ln_2.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.43.ln_2.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.43.mlp.c_fc.bias": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.43.mlp.c_fc.weight": "pytorch_model-00009-of-00010.bin",
+    "transformer.visual.transformer.resblocks.43.mlp.c_proj.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.43.mlp.c_proj.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.44.attn.in_proj.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.44.attn.in_proj.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.44.attn.out_proj.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.44.attn.out_proj.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.44.ln_1.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.44.ln_1.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.44.ln_2.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.44.ln_2.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.44.mlp.c_fc.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.44.mlp.c_fc.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.44.mlp.c_proj.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.44.mlp.c_proj.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.45.attn.in_proj.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.45.attn.in_proj.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.45.attn.out_proj.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.45.attn.out_proj.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.45.ln_1.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.45.ln_1.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.45.ln_2.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.45.ln_2.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.45.mlp.c_fc.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.45.mlp.c_fc.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.45.mlp.c_proj.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.45.mlp.c_proj.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.46.attn.in_proj.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.46.attn.in_proj.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.46.attn.out_proj.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.46.attn.out_proj.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.46.ln_1.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.46.ln_1.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.46.ln_2.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.46.ln_2.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.46.mlp.c_fc.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.46.mlp.c_fc.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.46.mlp.c_proj.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.46.mlp.c_proj.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.47.attn.in_proj.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.47.attn.in_proj.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.47.attn.out_proj.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.47.attn.out_proj.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.47.ln_1.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.47.ln_1.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.47.ln_2.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.47.ln_2.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.47.mlp.c_fc.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.47.mlp.c_fc.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.47.mlp.c_proj.bias": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.47.mlp.c_proj.weight": "pytorch_model-00010-of-00010.bin",
+    "transformer.visual.transformer.resblocks.5.attn.in_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.5.attn.in_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.5.attn.out_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.5.attn.out_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.5.ln_1.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.5.ln_1.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.5.ln_2.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.5.ln_2.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.5.mlp.c_fc.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.5.mlp.c_fc.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.5.mlp.c_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.5.mlp.c_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.6.attn.in_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.6.attn.in_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.6.attn.out_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.6.attn.out_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.6.ln_1.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.6.ln_1.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.6.ln_2.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.6.ln_2.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.6.mlp.c_fc.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.6.mlp.c_fc.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.6.mlp.c_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.6.mlp.c_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.7.attn.in_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.7.attn.in_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.7.attn.out_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.7.attn.out_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.7.ln_1.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.7.ln_1.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.7.ln_2.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.7.ln_2.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.7.mlp.c_fc.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.7.mlp.c_fc.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.7.mlp.c_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.7.mlp.c_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.8.attn.in_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.8.attn.in_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.8.attn.out_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.8.attn.out_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.8.ln_1.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.8.ln_1.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.8.ln_2.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.8.ln_2.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.8.mlp.c_fc.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.8.mlp.c_fc.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.8.mlp.c_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.8.mlp.c_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.9.attn.in_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.9.attn.in_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.9.attn.out_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.9.attn.out_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.9.ln_1.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.9.ln_1.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.9.ln_2.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.9.ln_2.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.9.mlp.c_fc.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.9.mlp.c_fc.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.9.mlp.c_proj.bias": "pytorch_model-00008-of-00010.bin",
+    "transformer.visual.transformer.resblocks.9.mlp.c_proj.weight": "pytorch_model-00008-of-00010.bin",
+    "transformer.wte.weight": "pytorch_model-00001-of-00010.bin"
+  }
+}

Qwen-VL-Chat/qwen_generation_utils.py

@@ -0,0 +1,420 @@
+# Copyright (c) Alibaba Cloud.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""Generation support."""
+
+from typing import Tuple, List, Union, Iterable
+
+import numpy as np
+import torch
+import torch.nn.functional as F
+from transformers import PreTrainedTokenizer
+from transformers import logging
+from transformers.generation import LogitsProcessor
+
+logger = logging.get_logger(__name__)
+
+# Types.
+HistoryType = List[Tuple[str, str]]
+TokensType = List[int]
+BatchTokensType = List[List[int]]
+
+
+def pad_batch(batch: BatchTokensType, pad_id: int, seq_length: int) -> BatchTokensType:
+    for tokens in batch:
+        context_length = len(tokens)
+        if context_length < seq_length:
+            tokens.extend([pad_id] * (seq_length - context_length))
+    return batch
+
+
+def get_ltor_masks_and_position_ids(
+    data,
+    eod_token,
+    reset_position_ids,
+    reset_attention_mask,
+    eod_mask_loss,
+):
+    """Build masks and position id for left to right model."""
+
+    # Extract batch size and sequence length.
+    micro_batch_size, seq_length = data.size()
+
+    # Attention mask (lower triangular).
+    if reset_attention_mask:
+        att_mask_batch = micro_batch_size
+    else:
+        att_mask_batch = 1
+    attention_mask = torch.tril(
+        torch.ones((att_mask_batch, seq_length, seq_length), device=data.device)
+    ).view(att_mask_batch, 1, seq_length, seq_length)
+
+    # Loss mask.
+    loss_mask = torch.ones(data.size(), dtype=torch.float, device=data.device)
+    if eod_mask_loss:
+        loss_mask[data == eod_token] = 0.0
+
+    # Position ids.
+    position_ids = torch.arange(seq_length, dtype=torch.long, device=data.device)
+    position_ids = position_ids.unsqueeze(0).expand_as(data)
+    # We need to clone as the ids will be modifed based on batch index.
+    if reset_position_ids:
+        position_ids = position_ids.clone()
+
+    if reset_position_ids or reset_attention_mask:
+        # Loop through the batches:
+        for b in range(micro_batch_size):
+
+            # Find indecies where EOD token is.
+            eod_index = position_ids[b, data[b] == eod_token]
+            # Detach indecies from positions if going to modify positions.
+            if reset_position_ids:
+                eod_index = eod_index.clone()
+
+            # Loop through EOD indecies:
+            prev_index = 0
+            for j in range(eod_index.size()[0]):
+                i = eod_index[j]
+                # Mask attention loss.
+                if reset_attention_mask:
+                    attention_mask[b, 0, (i + 1) :, : (i + 1)] = 0
+                # Reset positions.
+                if reset_position_ids:
+                    position_ids[b, (i + 1) :] -= i + 1 - prev_index
+                    prev_index = i + 1
+
+    # Convert attention mask to binary:
+    attention_mask = attention_mask < 0.5
+
+    return attention_mask, loss_mask, position_ids
+
+
+def get_batch(context_tokens: torch.LongTensor, eod_id: int):
+    """Generate batch from context tokens."""
+    # Move to GPU.
+    tokens = context_tokens.contiguous().to(context_tokens.device)
+    # Get the attention mask and postition ids.
+    attention_mask, _, position_ids = get_ltor_masks_and_position_ids(
+        tokens,
+        eod_id,
+        reset_position_ids=False,
+        reset_attention_mask=False,
+        eod_mask_loss=False,
+    )
+    return tokens, attention_mask, position_ids
+
+
+def get_stop_words_ids(chat_format, tokenizer):
+    if chat_format == "raw":
+        stop_words_ids = [tokenizer.encode("Human:"), [tokenizer.eod_id]]
+    elif chat_format == "chatml":
+        stop_words_ids = [[tokenizer.im_end_id], [tokenizer.im_start_id]]
+    else:
+        raise NotImplementedError(f"Unknown chat format {chat_format!r}")
+    return stop_words_ids
+
+
+def make_context(
+    tokenizer: PreTrainedTokenizer,
+    query: str,
+    history: List[Tuple[str, str]] = None,
+    system: str = "",
+    max_window_size: int = 6144,
+    chat_format: str = "chatml",
+):
+    if history is None:
+        history = []
+
+    if chat_format == "chatml":
+        im_start, im_end = "<|im_start|>", "<|im_end|>"
+        im_start_tokens = [tokenizer.im_start_id]
+        im_end_tokens = [tokenizer.im_end_id]
+        nl_tokens = tokenizer.encode("\n")
+
+        def _tokenize_str(role, content):
+            return f"{role}\n{content}", tokenizer.encode(
+                role, allowed_special=set(tokenizer.IMAGE_ST)
+            ) + nl_tokens + tokenizer.encode(content, allowed_special=set(tokenizer.IMAGE_ST))
+
+        system_text, system_tokens_part = _tokenize_str("system", system)
+        system_tokens = im_start_tokens + system_tokens_part + im_end_tokens
+
+        raw_text = ""
+        context_tokens = []
+
+        for turn_query, turn_response in reversed(history):
+            query_text, query_tokens_part = _tokenize_str("user", turn_query)
+            query_tokens = im_start_tokens + query_tokens_part + im_end_tokens
+            if turn_response is not None:
+                response_text, response_tokens_part = _tokenize_str(
+                    "assistant", turn_response
+                )
+                response_tokens = im_start_tokens + response_tokens_part + im_end_tokens
+
+                next_context_tokens = nl_tokens + query_tokens + nl_tokens + response_tokens
+                prev_chat = (
+                    f"\n{im_start}{query_text}{im_end}\n{im_start}{response_text}{im_end}"
+                )
+            else:
+                next_context_tokens = nl_tokens + query_tokens + nl_tokens
+                prev_chat = f"\n{im_start}{query_text}{im_end}\n"
+
+            current_context_size = (
+                len(system_tokens) + len(next_context_tokens) + len(context_tokens)
+            )
+            if current_context_size < max_window_size:
+                context_tokens = next_context_tokens + context_tokens
+                raw_text = prev_chat + raw_text
+            else:
+                break
+
+        context_tokens = system_tokens + context_tokens
+        raw_text = f"{im_start}{system_text}{im_end}" + raw_text
+        context_tokens += (
+            nl_tokens
+            + im_start_tokens
+            + _tokenize_str("user", query)[1]
+            + im_end_tokens
+            + nl_tokens
+            + im_start_tokens
+            + tokenizer.encode("assistant")
+            + nl_tokens
+        )
+        raw_text += f"\n{im_start}user\n{query}{im_end}\n{im_start}assistant\n"
+
+    elif chat_format == "raw":
+        raw_text = query
+        context_tokens = tokenizer.encode(raw_text)
+    else:
+        raise NotImplementedError(f"Unknown chat format {chat_format!r}")
+
+    return raw_text, context_tokens
+
+
+def _decode_default(
+    tokens: List[int],
+    *,
+    stop_words: List[str],
+    eod_words: List[str],
+    tokenizer: PreTrainedTokenizer,
+    raw_text_len: int,
+    verbose: bool = False,
+    return_end_reason: bool = False,
+    errors: str='replace',
+):
+    trim_decode_tokens = tokenizer.decode(tokens, errors=errors)[raw_text_len:]
+    if verbose:
+        print("\nRaw Generate: ", trim_decode_tokens)
+
+    end_reason = f"Gen length {len(tokens)}"
+    for stop_word in stop_words:
+        trim_decode_tokens = trim_decode_tokens.replace(stop_word, "").strip()
+    for eod_word in eod_words:
+        if eod_word in trim_decode_tokens:
+            end_reason = f"Gen {eod_word!r}"
+        trim_decode_tokens = trim_decode_tokens.split(eod_word)[0]
+    trim_decode_tokens = trim_decode_tokens.strip()
+    if verbose:
+        print("\nEnd Reason:", end_reason)
+        print("\nGenerate: ", trim_decode_tokens)
+
+    if return_end_reason:
+        return trim_decode_tokens, end_reason
+    else:
+        return trim_decode_tokens
+
+
+def _decode_chatml(
+    tokens: List[int],
+    *,
+    stop_words: List[str],
+    eod_token_ids: List[int],
+    tokenizer: PreTrainedTokenizer,
+    raw_text_len: int,
+    context_length: int,
+    verbose: bool = False,
+    return_end_reason: bool = False,
+    errors: str='replace'
+):
+    end_reason = f"Gen length {len(tokens)}"
+    eod_token_idx = context_length
+    for eod_token_idx in range(context_length, len(tokens)):
+        if tokens[eod_token_idx] in eod_token_ids:
+            end_reason = f"Gen {tokenizer.decode([tokens[eod_token_idx]])!r}"
+            break
+
+    trim_decode_tokens = tokenizer.decode(tokens[:eod_token_idx], errors=errors)[raw_text_len:]
+    if verbose:
+        print("\nRaw Generate w/o EOD:", tokenizer.decode(tokens, errors=errors)[raw_text_len:])
+        print("\nRaw Generate:", trim_decode_tokens)
+        print("\nEnd Reason:", end_reason)
+    for stop_word in stop_words:
+        trim_decode_tokens = trim_decode_tokens.replace(stop_word, "").strip()
+    trim_decode_tokens = trim_decode_tokens.strip()
+    if verbose:
+        print("\nGenerate:", trim_decode_tokens)
+
+    if return_end_reason:
+        return trim_decode_tokens, end_reason
+    else:
+        return trim_decode_tokens
+
+
+def decode_tokens(
+    tokens: Union[torch.LongTensor, TokensType],
+    tokenizer: PreTrainedTokenizer,
+    raw_text_len: int,
+    context_length: int,
+    chat_format: str,
+    verbose: bool = False,
+    return_end_reason: bool = False,
+    errors: str="replace",
+) -> str:
+    if torch.is_tensor(tokens):
+        tokens = tokens.cpu().numpy().tolist()
+
+    if chat_format == "chatml":
+        return _decode_chatml(
+            tokens,
+            stop_words=[],
+            eod_token_ids=[tokenizer.im_start_id, tokenizer.im_end_id],
+            tokenizer=tokenizer,
+            raw_text_len=raw_text_len,
+            context_length=context_length,
+            verbose=verbose,
+            return_end_reason=return_end_reason,
+            errors=errors,
+        )
+    elif chat_format == "raw":
+        return _decode_default(
+            tokens,
+            stop_words=["<|endoftext|>"],
+            eod_words=["<|endoftext|>"],
+            tokenizer=tokenizer,
+            raw_text_len=raw_text_len,
+            verbose=verbose,
+            return_end_reason=return_end_reason,
+            errors=errors,
+        )
+    else:
+        raise NotImplementedError(f"Unknown chat format {chat_format!r}")
+
+
+class StopWordsLogitsProcessor(LogitsProcessor):
+    """
+    :class:`transformers.LogitsProcessor` that enforces that when specified sequences appear, stop geration.
+
+    Args:
+        stop_words_ids (:obj:`List[List[int]]`):
+            List of list of token ids of stop ids. In order to get the tokens of the words
+            that should not appear in the generated text, use :obj:`tokenizer(bad_word,
+            add_prefix_space=True).input_ids`.
+        eos_token_id (:obj:`int`):
+            The id of the `end-of-sequence` token.
+    """
+
+    def __init__(self, stop_words_ids: Iterable[Iterable[int]], eos_token_id: int):
+
+        if not isinstance(stop_words_ids, List) or len(stop_words_ids) == 0:
+            raise ValueError(
+                f"`stop_words_ids` has to be a non-emtpy list, but is {stop_words_ids}."
+            )
+        if any(not isinstance(bad_word_ids, list) for bad_word_ids in stop_words_ids):
+            raise ValueError(
+                f"`stop_words_ids` has to be a list of lists, but is {stop_words_ids}."
+            )
+        if any(
+            any(
+                (not isinstance(token_id, (int, np.integer)) or token_id < 0)
+                for token_id in stop_word_ids
+            )
+            for stop_word_ids in stop_words_ids
+        ):
+            raise ValueError(
+                f"Each list in `stop_words_ids` has to be a list of positive integers, but is {stop_words_ids}."
+            )
+
+        self.stop_words_ids = list(
+            filter(
+                lambda bad_token_seq: bad_token_seq != [eos_token_id], stop_words_ids
+            )
+        )
+        self.eos_token_id = eos_token_id
+        for stop_token_seq in self.stop_words_ids:
+            assert (
+                len(stop_token_seq) > 0
+            ), "Stop words token sequences {} cannot have an empty list".format(
+                stop_words_ids
+            )
+
+    def __call__(
+        self, input_ids: torch.LongTensor, scores: torch.FloatTensor
+    ) -> torch.FloatTensor:
+        stopped_samples = self._calc_stopped_samples(input_ids)
+        for i, should_stop in enumerate(stopped_samples):
+            if should_stop:
+                scores[i, self.eos_token_id] = float(2**15)
+        return scores
+
+    def _tokens_match(self, prev_tokens: torch.LongTensor, tokens: List[int]) -> bool:
+        if len(tokens) == 0:
+            # if bad word tokens is just one token always ban it
+            return True
+        elif len(tokens) > len(prev_tokens):
+            # if bad word tokens are longer then prev input_ids they can't be equal
+            return False
+        elif prev_tokens[-len(tokens) :].tolist() == tokens:
+            # if tokens match
+            return True
+        else:
+            return False
+
+    def _calc_stopped_samples(self, prev_input_ids: Iterable[int]) -> Iterable[int]:
+        stopped_samples = []
+        for prev_input_ids_slice in prev_input_ids:
+            match = False
+            for stop_token_seq in self.stop_words_ids:
+                if self._tokens_match(prev_input_ids_slice, stop_token_seq):
+                    # if tokens do not match continue
+                    match = True
+                    break
+            stopped_samples.append(match)
+
+        return stopped_samples
+
+
+def top_k_logits(logits, top_k=0, top_p=0.0, filter_value=-float("Inf")):
+    """This function has been mostly taken from huggingface conversational
+    ai code at
+        https://medium.com/huggingface/how-to-build-a-state-of-the-art-
+             conversational-ai-with-transfer-learning-2d818ac26313"""
+
+    if top_k > 0:
+        # Remove all tokens with a probability less than the
+        # last token of the top-k
+        indices_to_remove = logits < torch.topk(logits, top_k)[0][..., -1, None]
+        logits[indices_to_remove] = filter_value
+
+    if top_p > 0.0:
+        # Cconvert to 1D
+        sorted_logits, sorted_indices = torch.sort(logits, descending=True, dim=-1)
+        cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)
+
+        # Remove tokens with cumulative probability above the threshold
+        sorted_indices_to_remove = cumulative_probs > top_p
+        # Shift the indices to the right to keep also the first token
+        # above the threshold
+        sorted_indices_to_remove[..., 1:] = sorted_indices_to_remove[..., :-1].clone()
+        sorted_indices_to_remove[..., 0] = 0
+        for i in range(sorted_indices.size(0)):
+            indices_to_remove = sorted_indices[i][sorted_indices_to_remove[i]]
+            logits[i][indices_to_remove] = filter_value
+
+    return logits
+
+
+def switch(val1, val2, boolean):
+    boolean = boolean.type_as(val1)
+    return (1 - boolean) * val1 + boolean * val2

Qwen-VL-Chat/tokenization_qwen.py

@@ -0,0 +1,598 @@
+# Copyright (c) Alibaba Cloud.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""Tokenization classes for QWen."""
+
+import base64
+import logging
+import os
+import requests
+import unicodedata
+from typing import Collection, Dict, List, Set, Tuple, Union, Any, Callable, Optional
+
+import tiktoken
+import numpy as np
+from PIL import Image
+from PIL import ImageFont
+from PIL import ImageDraw
+from transformers import PreTrainedTokenizer, AddedToken
+from transformers.utils import try_to_load_from_cache
+
+import matplotlib.colors as mcolors
+from matplotlib.font_manager import FontProperties
+
+logger = logging.getLogger(__name__)
+
+
+VOCAB_FILES_NAMES = {"vocab_file": "qwen.tiktoken", "ttf": "SimSun.ttf"}
+FONT_PATH = try_to_load_from_cache("Qwen/Qwen-VL-Chat", "SimSun.ttf")
+if FONT_PATH is None:
+    if not os.path.exists("SimSun.ttf"):
+        ttf = requests.get("https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/SimSun.ttf")
+        open("SimSun.ttf", "wb").write(ttf.content)
+    FONT_PATH = "SimSun.ttf"
+
+PAT_STR = r"""(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+"""
+ENDOFTEXT = "<|endoftext|>"
+IMSTART = "<|im_start|>"
+IMEND = "<|im_end|>"
+# as the default behavior is changed to allow special tokens in
+# regular texts, the surface forms of special tokens need to be
+# as different as possible to minimize the impact
+EXTRAS = tuple((f"<|extra_{i}|>" for i in range(205)))
+SPECIAL_TOKENS = (
+    ENDOFTEXT,
+    IMSTART,
+    IMEND,
+) + EXTRAS
+IMG_TOKEN_SPAN = 256
+
+
+def _load_tiktoken_bpe(tiktoken_bpe_file: str) -> Dict[bytes, int]:
+    with open(tiktoken_bpe_file, "rb") as f:
+        contents = f.read()
+    return {
+        base64.b64decode(token): int(rank)
+        for token, rank in (line.split() for line in contents.splitlines() if line)
+    }
+
+def _list_find(
+    input_list: List[Any],
+    candidates: Tuple[Any],
+    start: int = 0,
+):
+    for i in range(start, len(input_list)):
+        if input_list[i] in candidates:
+            return i
+    return -1
+
+def _replace_closed_tag(
+    input_tokens: List[Any],
+    start_tags: Union[Any, Tuple[Any]],
+    end_tags: Union[Any, Tuple[Any]],
+    inclusive_replace_func: Callable,
+    exclusive_replace_func: Callable = lambda x: x,
+):
+    if isinstance(start_tags, (str, int)):
+        start_tags = (start_tags,)
+    if isinstance(end_tags, (str, int)):
+        end_tags = (end_tags,)
+    assert len(start_tags) == len(end_tags)
+
+    output_tokens = []
+    end = 0
+    while True:
+        start = _list_find(input_tokens, start_tags, end)
+        if start == -1:
+            break
+        output_tokens.extend(exclusive_replace_func(input_tokens[end : start]))
+        tag_idx = start_tags.index(input_tokens[start])
+        end = _list_find(input_tokens, (end_tags[tag_idx],), start)
+        if end == -1:
+            raise ValueError("Unclosed image token")
+        output_tokens.extend(inclusive_replace_func(input_tokens[start : end + 1]))
+        end += 1
+    output_tokens.extend(exclusive_replace_func(input_tokens[end : ]))
+    return output_tokens
+
+class QWenTokenizer(PreTrainedTokenizer):
+    """QWen tokenizer."""
+
+    vocab_files_names = VOCAB_FILES_NAMES
+
+    def __init__(
+        self,
+        vocab_file,
+        errors="replace",
+        image_start_tag='<img>',
+        image_end_tag='</img>',
+        image_pad_tag='<imgpad>',
+        ref_start_tag='<ref>',
+        ref_end_tag='</ref>',
+        box_start_tag='<box>',
+        box_end_tag='</box>',
+        quad_start_tag='<quad>',
+        quad_end_tag='</quad>',
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.image_start_tag = image_start_tag
+        self.image_end_tag = image_end_tag
+        self.image_pad_tag = image_pad_tag
+        self.ref_start_tag = ref_start_tag
+        self.ref_end_tag = ref_end_tag
+        self.box_start_tag = box_start_tag
+        self.box_end_tag = box_end_tag
+        self.quad_start_tag = quad_start_tag
+        self.quad_end_tag = quad_end_tag
+        self.IMAGE_ST = (
+            ref_start_tag, ref_end_tag,
+            box_start_tag, box_end_tag,
+            quad_start_tag, quad_end_tag,
+            image_start_tag, image_end_tag,
+            image_pad_tag
+        )
+
+        self.errors = errors  # how to handle errors in decoding
+
+        self.mergeable_ranks = _load_tiktoken_bpe(vocab_file)  # type: dict[bytes, int]
+        self.special_tokens = {
+            token: index
+            for index, token in enumerate(
+                SPECIAL_TOKENS + self.IMAGE_ST, start=len(self.mergeable_ranks)
+            )
+        }
+        self.img_start_id = self.special_tokens[self.image_start_tag]
+        self.img_end_id = self.special_tokens[self.image_end_tag]
+        self.img_pad_id = self.special_tokens[self.image_pad_tag]
+        self.ref_start_id = self.special_tokens[self.ref_start_tag]
+        self.ref_end_id = self.special_tokens[self.ref_end_tag]
+        self.box_start_id = self.special_tokens[self.box_start_tag]
+        self.box_end_id = self.special_tokens[self.box_end_tag]
+        self.quad_start_id = self.special_tokens[self.quad_start_tag]
+        self.quad_end_id = self.special_tokens[self.quad_end_tag]
+        self.image_special_tokens = set([
+            self.ref_start_id, self.ref_end_id, self.box_start_id, self.box_end_id,
+            self.quad_start_id, self.quad_end_id,
+        ])
+
+        enc = tiktoken.Encoding(
+            "Qwen",
+            pat_str=PAT_STR,
+            mergeable_ranks=self.mergeable_ranks,
+            special_tokens=self.special_tokens,
+        )
+        assert (
+            len(self.mergeable_ranks) + len(self.special_tokens) == enc.n_vocab
+        ), f"{len(self.mergeable_ranks) + len(self.special_tokens)} != {enc.n_vocab} in encoding"
+
+        self.decoder = {
+            v: k for k, v in self.mergeable_ranks.items()
+        }  # type: dict[int, bytes|str]
+        self.decoder.update({v: k for k, v in self.special_tokens.items()})
+
+        self.tokenizer = enc  # type: tiktoken.Encoding
+
+        self.eod_id = self.tokenizer.eot_token
+        self.im_start_id = self.special_tokens[IMSTART]
+        self.im_end_id = self.special_tokens[IMEND]
+
+    def __getstate__(self):
+        # for pickle lovers
+        state = self.__dict__.copy()
+        del state['tokenizer']
+        return state
+
+    def __setstate__(self, state):
+        # tokenizer is not python native; don't pass it; rebuild it
+        self.__dict__.update(state)
+        enc = tiktoken.Encoding(
+            "Qwen",
+            pat_str=PAT_STR,
+            mergeable_ranks=self.mergeable_ranks,
+            special_tokens=self.special_tokens,
+        )
+        self.tokenizer = enc
+
+
+    def __len__(self) -> int:
+        return self.tokenizer.n_vocab
+
+    def get_vocab(self) -> Dict[bytes, int]:
+        return self.mergeable_ranks
+
+    def convert_tokens_to_ids(
+        self, tokens: Union[bytes, str, List[Union[bytes, str]]]
+    ) -> List[int]:
+        ids = []
+        if isinstance(tokens, (str, bytes)):
+            if tokens in self.special_tokens:
+                return self.special_tokens[tokens]
+            else:
+                return self.mergeable_ranks.get(tokens)
+        for token in tokens:
+            if token in self.special_tokens:
+                ids.append(self.special_tokens[token])
+            else:
+                ids.append(self.mergeable_ranks.get(token))
+        return ids
+
+    def _add_tokens(self, new_tokens: Union[List[str], List[AddedToken]], special_tokens: bool = False) -> int:
+        if not special_tokens and new_tokens:
+            raise ValueError('Adding regular tokens is not supported')
+        for token in new_tokens:
+            surface_form = token.content if isinstance(token, AddedToken) else token
+            if surface_form not in SPECIAL_TOKENS + self.IMAGE_ST:
+                raise ValueError('Adding unknown special tokens is not supported')
+        return 0
+
+    def save_vocabulary(self, save_directory: str, **kwargs) -> Tuple[str]:
+        """
+        Save only the vocabulary of the tokenizer (vocabulary).
+
+        Returns:
+            `Tuple(str)`: Paths to the files saved.
+        """
+        file_path = os.path.join(save_directory, "qwen.tiktoken")
+        with open(file_path, "w", encoding="utf8") as w:
+            for k, v in self.mergeable_ranks.items():
+                line = base64.b64encode(k).decode("utf8") + " " + str(v) + "\n"
+                w.write(line)
+        return (file_path,)
+
+    def tokenize(
+        self,
+        text: str,
+        allowed_special: Union[Set, str] = "all",
+        disallowed_special: Union[Collection, str] = (),
+        **kwargs,
+    ) -> List[Union[bytes, str]]:
+        """
+        Converts a string in a sequence of tokens.
+
+        Args:
+            text (`str`):
+                The sequence to be encoded.
+            allowed_special (`Literal["all"]` or `set`):
+                The surface forms of the tokens to be encoded as special tokens in regular texts.
+                Default to "all".
+            disallowed_special (`Literal["all"]` or `Collection`):
+                The surface forms of the tokens that should not be in regular texts and trigger errors.
+                Default to an empty tuple.
+
+            kwargs (additional keyword arguments, *optional*):
+                Will be passed to the underlying model specific encode method.
+
+        Returns:
+            `List[bytes|str]`: The list of tokens.
+        """
+        tokens = []
+        text = unicodedata.normalize("NFC", text)
+
+        # this implementation takes a detour: text -> token id -> token surface forms
+        for t in self.tokenizer.encode(
+            text, allowed_special=allowed_special, disallowed_special=disallowed_special
+        ):
+            tokens.append(self.decoder[t])
+
+        def _encode_imgurl(img_tokens):
+            assert img_tokens[0] == self.image_start_tag and img_tokens[-1] == self.image_end_tag
+            img_tokens = img_tokens[1:-1]
+            img_url = b''.join(img_tokens)
+            out_img_tokens = list(map(self.decoder.get, img_url))
+            if len(out_img_tokens) > IMG_TOKEN_SPAN:
+                raise ValueError("The content in {}..{} is too long".format(
+                    self.image_start_tag, self.image_end_tag))
+            out_img_tokens.extend([self.image_pad_tag] * (IMG_TOKEN_SPAN - len(out_img_tokens)))
+            out_img_tokens = [self.image_start_tag] + out_img_tokens + [self.image_end_tag]
+            return out_img_tokens
+
+        return _replace_closed_tag(tokens, self.image_start_tag, self.image_end_tag, _encode_imgurl)
+
+    def convert_tokens_to_string(self, tokens: List[Union[bytes, str]]) -> str:
+        """
+        Converts a sequence of tokens in a single string.
+        """
+        text = ""
+        temp = b""
+        for t in tokens:
+            if isinstance(t, str):
+                if temp:
+                    text += temp.decode("utf-8", errors=self.errors)
+                    temp = b""
+                text += t
+            elif isinstance(t, bytes):
+                temp += t
+            else:
+                raise TypeError("token should only be of type types or str")
+        if temp:
+            text += temp.decode("utf-8", errors=self.errors)
+        return text
+
+    @property
+    def vocab_size(self):
+        return self.tokenizer.n_vocab
+
+    def _convert_id_to_token(self, index: int) -> Union[bytes, str]:
+        """Converts an id to a token, special tokens included"""
+        if index in self.decoder:
+            return self.decoder[index]
+        raise ValueError("unknown ids")
+
+    def _convert_token_to_id(self, token: Union[bytes, str]) -> int:
+        """Converts a token to an id using the vocab, special tokens included"""
+        if token in self.special_tokens:
+            return self.special_tokens[token]
+        if token in self.mergeable_ranks:
+            return self.mergeable_ranks[token]
+        raise ValueError("unknown token")
+
+    def _tokenize(self, text: str, **kwargs):
+        """
+        Converts a string in a sequence of tokens (string), using the tokenizer. Split in words for word-based
+        vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces).
+
+        Do NOT take care of added tokens.
+        """
+        raise NotImplementedError
+
+    def _decode(
+        self,
+        token_ids: Union[int, List[int]],
+        skip_special_tokens: bool = False,
+        errors: str = None,
+        **kwargs,
+    ) -> str:
+        if isinstance(token_ids, int):
+            token_ids = [token_ids]
+
+        def _decode_imgurl(img_token_ids):
+            assert img_token_ids[0] == self.img_start_id and img_token_ids[-1] == self.img_end_id
+            img_token_ids = img_token_ids[1:-1]
+            img_token_ids = img_token_ids[ : img_token_ids.index(self.img_pad_id)]
+            img_url = bytes(img_token_ids).decode('utf-8')
+            return [self.img_start_id] + self.tokenizer.encode(img_url) + [self.img_end_id]
+
+        token_ids = _replace_closed_tag(token_ids, self.img_start_id, self.img_end_id, _decode_imgurl)
+
+        if skip_special_tokens:
+            if kwargs.get('keep_image_special', False):
+                token_ids = [i for i in token_ids if i < self.eod_id 
+                    or i in self.image_special_tokens]
+            else:
+                token_ids = [i for i in token_ids if i < self.eod_id]
+        return self.tokenizer.decode(token_ids, errors=errors or self.errors)
+
+    def to_list_format(self, text: str):
+        text = unicodedata.normalize("NFC", text)
+        token_ids = self.tokenizer.encode(
+            text, allowed_special=set(self.IMAGE_ST + (ENDOFTEXT,)))
+
+        def _encode_vl_info(tokens):
+            if len(tokens) == 0:
+                return []
+            if tokens[0] == self.img_start_id and tokens[-1] == self.img_end_id:
+                key = 'image'
+            elif tokens[0] == self.ref_start_id and tokens[-1] == self.ref_end_id:
+                key = 'ref'
+            elif tokens[0] == self.box_start_id and tokens[-1] == self.box_end_id:
+                key = 'box'
+            elif tokens[0] == self.quad_start_id and tokens[-1] == self.quad_end_id:
+                key = 'quad'
+            else:
+                _tobytes = lambda x: x.encode('utf-8') if isinstance(x, str) else x
+                return [{'text': b''.join(map(_tobytes, map(self.decoder.get, tokens))).decode('utf-8')}]
+            _tobytes = lambda x: x.encode('utf-8') if isinstance(x, str) else x
+            val = b''.join(map(_tobytes, map(self.decoder.get, tokens[1:-1]))).decode('utf-8')
+            return [{key: val}]
+
+        return _replace_closed_tag(
+            token_ids,
+            (self.img_start_id, self.ref_start_id, self.box_start_id, self.quad_start_id),
+            (self.img_end_id, self.ref_end_id, self.box_end_id, self.quad_end_id),
+            _encode_vl_info,
+            _encode_vl_info,
+        )
+
+    def from_list_format(self, list_format: List[Dict]):
+        text = ''
+        num_images = 0
+        for ele in list_format:
+            if 'image' in ele:
+                num_images += 1
+                text += f'Picture {num_images}: '
+                text += self.image_start_tag + ele['image'] + self.image_end_tag
+                text += '\n'
+            elif 'text' in ele:
+                text += ele['text']
+            elif 'box' in ele:
+                if 'ref' in ele:
+                    text += self.ref_start_tag + ele['ref'] + self.ref_end_tag
+                for box in ele['box']:
+                    text += self.box_start_tag + '(%d,%d),(%d,%d)' % (box[0], box[1], box[2], box[3]) + self.box_end_tag
+            else:
+                raise ValueError("Unsupport element: " + str(ele))
+        return text
+
+    def _fetch_latest_picture(self, response, history):
+        if history is None:
+            history = []
+        _history = history + [(response, None)]
+        for q, r in _history[::-1]:
+            for ele in self.to_list_format(q)[::-1]:
+                if 'image' in ele:
+                    return ele['image']
+        return None
+
+    def _fetch_all_box_with_ref(self, text):
+        list_format = self.to_list_format(text)
+        output = []
+        for i, ele in enumerate(list_format):
+            if 'box' in ele:
+                bbox = tuple(map(int, ele['box'].replace('(', '').replace(')', '').split(',')))
+                assert len(bbox) == 4
+                output.append({'box': bbox})
+                if i > 0 and 'ref' in list_format[i-1]:
+                    output[-1]['ref'] = list_format[i-1]['ref'].strip()
+        return output
+
+    def draw_bbox_on_latest_picture(
+        self,
+        response,
+        history=None,
+    ) -> Optional[Image.Image]:
+        image = self._fetch_latest_picture(response, history)
+        if image is None:
+            return None
+        if image.startswith("http://") or image.startswith("https://"):
+            image = Image.open(requests.get(image, stream=True).raw).convert("RGB")
+            h, w = image.height, image.width
+        else:
+            image = np.asarray(Image.open(image).convert("RGB"))
+            h, w = image.shape[0], image.shape[1]
+        visualizer = Visualizer(image)
+
+        boxes = self._fetch_all_box_with_ref(response)
+        if not boxes:
+            return None
+        color = random.choice([_ for _ in mcolors.TABLEAU_COLORS.keys()]) # init color
+        for box in boxes:
+            if 'ref' in box: # random new color for new refexps
+                color = random.choice([_ for _ in mcolors.TABLEAU_COLORS.keys()])
+            x1, y1, x2, y2 = box['box']
+            x1, y1, x2, y2 = (int(x1 / 1000 * w), int(y1 / 1000 * h), int(x2 / 1000 * w), int(y2 / 1000 * h))
+            visualizer.draw_box((x1, y1, x2, y2), alpha=1, edge_color=color)
+            if 'ref' in box:
+                visualizer.draw_text(box['ref'], (x1, y1), color=color, horizontal_alignment="left")
+        return visualizer.output
+
+
+import colorsys
+import logging
+import math
+import numpy as np
+import matplotlib as mpl
+import matplotlib.colors as mplc
+import matplotlib.figure as mplfigure
+import torch
+from matplotlib.backends.backend_agg import FigureCanvasAgg
+from PIL import Image
+import random
+
+logger = logging.getLogger(__name__)
+
+
+class VisImage:
+    def __init__(self, img, scale=1.0):
+        self.img = img
+        self.scale = scale
+        self.width, self.height = img.shape[1], img.shape[0]
+        self._setup_figure(img)
+
+    def _setup_figure(self, img):
+        fig = mplfigure.Figure(frameon=False)
+        self.dpi = fig.get_dpi()
+        # add a small 1e-2 to avoid precision lost due to matplotlib's truncation
+        # (https://github.com/matplotlib/matplotlib/issues/15363)
+        fig.set_size_inches(
+            (self.width * self.scale + 1e-2) / self.dpi,
+            (self.height * self.scale + 1e-2) / self.dpi,
+        )
+        self.canvas = FigureCanvasAgg(fig)
+        # self.canvas = mpl.backends.backend_cairo.FigureCanvasCairo(fig)
+        ax = fig.add_axes([0.0, 0.0, 1.0, 1.0])
+        ax.axis("off")
+        self.fig = fig
+        self.ax = ax
+        self.reset_image(img)
+
+    def reset_image(self, img):
+        img = img.astype("uint8")
+        self.ax.imshow(img, extent=(0, self.width, self.height, 0), interpolation="nearest")
+
+    def save(self, filepath):
+        self.fig.savefig(filepath)
+
+    def get_image(self):
+        canvas = self.canvas
+        s, (width, height) = canvas.print_to_buffer()
+
+        buffer = np.frombuffer(s, dtype="uint8")
+
+        img_rgba = buffer.reshape(height, width, 4)
+        rgb, alpha = np.split(img_rgba, [3], axis=2)
+        return rgb.astype("uint8")
+
+
+class Visualizer:
+    def __init__(self, img_rgb, metadata=None, scale=1.0):
+        self.img = np.asarray(img_rgb).clip(0, 255).astype(np.uint8)
+        self.font_path = FONT_PATH
+        self.output = VisImage(self.img, scale=scale)
+        self.cpu_device = torch.device("cpu")
+
+        # too small texts are useless, therefore clamp to 14
+        self._default_font_size = max(
+            np.sqrt(self.output.height * self.output.width) // 30, 15 // scale
+        )
+
+    def draw_text(
+        self,
+        text,
+        position,
+        *,
+        font_size=None,
+        color="g",
+        horizontal_alignment="center",
+        rotation=0,
+    ):
+        if not font_size:
+            font_size = self._default_font_size
+
+        # since the text background is dark, we don't want the text to be dark
+        color = np.maximum(list(mplc.to_rgb(color)), 0.2)
+        color[np.argmax(color)] = max(0.8, np.max(color))
+
+        x, y = position
+        self.output.ax.text(
+            x,
+            y,
+            text,
+            size=font_size * self.output.scale,
+            fontproperties=FontProperties(fname=self.font_path),
+            bbox={"facecolor": "black", "alpha": 0.8, "pad": 0.7, "edgecolor": "none"},
+            verticalalignment="top",
+            horizontalalignment=horizontal_alignment,
+            color=color,
+            zorder=10,
+            rotation=rotation,
+        )
+        return self.output
+
+    def draw_box(self, box_coord, alpha=0.5, edge_color="g", line_style="-"):
+        
+        x0, y0, x1, y1 = box_coord
+        width = x1 - x0
+        height = y1 - y0
+
+        linewidth = max(self._default_font_size / 4, 1)
+
+        self.output.ax.add_patch(
+            mpl.patches.Rectangle(
+                (x0, y0),
+                width,
+                height,
+                fill=False,
+                edgecolor=edge_color,
+                linewidth=linewidth * self.output.scale,
+                alpha=alpha,
+                linestyle=line_style,
+            )
+        )
+        return self.output
+
+    def get_output(self):
+        
+        return self.output

Qwen-VL-Chat/tokenizer_config.json

@@ -0,0 +1,10 @@
+{
+  "model_max_length": 8192,
+  "tokenizer_class": "QWenTokenizer",
+  "auto_map": {
+    "AutoTokenizer": [
+      "tokenization_qwen.QWenTokenizer",
+      null
+      ]
+  }
+}

Qwen-VL-Chat/visual.py

@@ -0,0 +1,426 @@
+# Copyright (c) Alibaba Cloud.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+from collections import OrderedDict
+import math
+import requests
+from io import BytesIO
+from functools import partial
+from PIL import Image
+from typing import Callable, Optional, Sequence, Tuple, List
+import numpy as np
+
+import torch
+from torch import nn
+from torch.nn import functional as F
+from torch.nn.init import trunc_normal_
+from torchvision import transforms
+from torchvision.transforms import InterpolationMode
+
+
+def get_abs_pos(abs_pos, tgt_size):
+    # abs_pos: L, C
+    # tgt_size: M
+    # return: M, C
+    src_size = int(math.sqrt(abs_pos.size(0)))
+    tgt_size = int(math.sqrt(tgt_size))
+    dtype = abs_pos.dtype
+
+    if src_size != tgt_size:
+        return F.interpolate(
+            abs_pos.float().reshape(1, src_size, src_size, -1).permute(0, 3, 1, 2),
+            size=(tgt_size, tgt_size),
+            mode="bicubic",
+            align_corners=False,
+        ).permute(0, 2, 3, 1).flatten(0, 2).to(dtype=dtype)
+    else:
+        return abs_pos
+
+# https://github.com/facebookresearch/mae/blob/efb2a8062c206524e35e47d04501ed4f544c0ae8/util/pos_embed.py#L20
+def get_2d_sincos_pos_embed(embed_dim, grid_size, cls_token=False):
+    """
+    grid_size: int of the grid height and width
+    return:
+    pos_embed: [grid_size*grid_size, embed_dim] or [1+grid_size*grid_size, embed_dim] (w/ or w/o cls_token)
+    """
+    grid_h = np.arange(grid_size, dtype=np.float32)
+    grid_w = np.arange(grid_size, dtype=np.float32)
+    grid = np.meshgrid(grid_w, grid_h)  # here w goes first
+    grid = np.stack(grid, axis=0)
+
+    grid = grid.reshape([2, 1, grid_size, grid_size])
+    pos_embed = get_2d_sincos_pos_embed_from_grid(embed_dim, grid)
+    if cls_token:
+        pos_embed = np.concatenate([np.zeros([1, embed_dim]), pos_embed], axis=0)
+    return pos_embed
+
+
+def get_2d_sincos_pos_embed_from_grid(embed_dim, grid):
+    assert embed_dim % 2 == 0
+
+    # use half of dimensions to encode grid_h
+    emb_h = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[0])  # (H*W, D/2)
+    emb_w = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[1])  # (H*W, D/2)
+
+    emb = np.concatenate([emb_h, emb_w], axis=1) # (H*W, D)
+    return emb
+
+
+def get_1d_sincos_pos_embed_from_grid(embed_dim, pos):
+    """
+    embed_dim: output dimension for each position
+    pos: a list of positions to be encoded: size (M,)
+    out: (M, D)
+    """
+    assert embed_dim % 2 == 0
+    omega = np.arange(embed_dim // 2, dtype=np.float32)
+    omega /= embed_dim / 2.
+    omega = 1. / 10000**omega  # (D/2,)
+
+    pos = pos.reshape(-1)  # (M,)
+    out = np.einsum('m,d->md', pos, omega)  # (M, D/2), outer product
+
+    emb_sin = np.sin(out) # (M, D/2)
+    emb_cos = np.cos(out) # (M, D/2)
+
+    emb = np.concatenate([emb_sin, emb_cos], axis=1)  # (M, D)
+    return emb
+
+
+class Resampler(nn.Module):
+    """
+    A 2D perceiver-resampler network with one cross attention layers by
+        (grid_size**2) learnable queries and 2d sincos pos_emb
+    Outputs:
+        A tensor with the shape of (grid_size**2, embed_dim)
+    """
+    def __init__(
+            self,
+            grid_size,
+            embed_dim,
+            num_heads,
+            kv_dim=None,
+            norm_layer=nn.LayerNorm
+    ):
+        super().__init__()
+        self.num_queries = grid_size ** 2
+        self.embed_dim = embed_dim
+        self.num_heads = num_heads
+
+        self.pos_embed = nn.Parameter(
+            torch.from_numpy(get_2d_sincos_pos_embed(embed_dim, grid_size)).float()
+        ).requires_grad_(False)
+
+        self.query = nn.Parameter(torch.zeros(self.num_queries, embed_dim))
+        trunc_normal_(self.query, std=.02)
+
+        if kv_dim is not None and kv_dim != embed_dim:
+            self.kv_proj = nn.Linear(kv_dim, embed_dim, bias=False)
+        else:
+            self.kv_proj = nn.Identity()
+
+        self.attn = nn.MultiheadAttention(embed_dim, num_heads)
+        self.ln_q = norm_layer(embed_dim)
+        self.ln_kv = norm_layer(embed_dim)
+        
+        # self.apply(self._init_weights)
+
+    def _init_weights(self, m):
+        if isinstance(m, nn.Linear):
+            trunc_normal_(m.weight, std=.02)
+            if isinstance(m, nn.Linear) and m.bias is not None:
+                nn.init.constant_(m.bias, 0)
+        elif isinstance(m, nn.LayerNorm):
+            nn.init.constant_(m.bias, 0)
+            nn.init.constant_(m.weight, 1.0)
+
+    def forward(self, x, attn_mask=None):
+
+        pos_embed = get_abs_pos(self.pos_embed, x.size(1))
+
+        x = self.kv_proj(x)
+        x = self.ln_kv(x).permute(1, 0, 2)
+
+        N = x.shape[1]
+        q = self.ln_q(self.query)
+        out = self.attn(
+            self._repeat(q, N) + self.pos_embed.unsqueeze(1),
+            x + pos_embed.unsqueeze(1),
+            x,
+            attn_mask=attn_mask)[0]
+        return out.permute(1, 0, 2)
+
+    def _repeat(self, query, N: int):
+        return query.unsqueeze(1).repeat(1, N, 1)
+
+
+class VisualAttention(nn.Module):
+    """self-attention layer class.
+
+    Self-attention layer takes input with size [s, b, h]
+    and returns output of the same size.
+    """
+
+    def __init__(self, embed_dim, num_heads,
+                 bias=True, kdim=None, vdim=None):
+        super(VisualAttention, self).__init__()
+        self.embed_dim = embed_dim
+        self.kdim = kdim if kdim is not None else embed_dim
+        self.vdim = vdim if vdim is not None else embed_dim
+        self._qkv_same_embed_dim = self.kdim == embed_dim and self.vdim == embed_dim
+
+        self.num_heads = num_heads
+
+        # Per attention head and per partition values.
+        assert embed_dim % num_heads == 0
+        self.hidden_size_per_attention_head = embed_dim // num_heads
+        self.num_attention_heads_per_partition = num_heads
+        self.hidden_size_per_partition = embed_dim
+
+        # Strided linear layer.
+        assert self._qkv_same_embed_dim, 'Only Support SelfAttention Currently'
+        self.in_proj = nn.Linear(embed_dim, 3 * embed_dim)
+        self.out_proj = nn.Linear(embed_dim, embed_dim)
+        self.norm_factor = math.sqrt(self.hidden_size_per_attention_head)
+
+    def forward(self, query, key, value, attn_mask = None):
+        # query/key/value: [sq, b, h]
+        sq, b, _ = query.size()
+
+        assert torch.allclose(query, key), 'Only Support Self-Attention Currently'
+        sk = sq
+        mixed_x_layer = self.in_proj(query)
+
+        # [sq, b, (np * 3 * hn)] --> [sq, b, np, 3 * hn]
+        new_tensor_shape = mixed_x_layer.size()[:-1] + \
+            (self.num_attention_heads_per_partition,
+             3 * self.hidden_size_per_attention_head)
+        mixed_x_layer = mixed_x_layer.view(*new_tensor_shape)
+
+        # [sq, b, np, 3 * hn] --> 3 [sq, b, np, hn]
+        query_layer, key_layer, value_layer = mixed_x_layer.split(
+            self.hidden_size_per_attention_head, dim=-1)
+
+        # [sq, b, np, hn] -> [sq, b * np, hn]
+        query_layer = query_layer.view(sq,
+            b * self.num_attention_heads_per_partition,
+            self.hidden_size_per_attention_head).transpose(0, 1)
+        # [sk, b, np, hn] -> [sk, b * np, hn]
+        key_layer = key_layer.view(sk,
+            b * self.num_attention_heads_per_partition,
+            self.hidden_size_per_attention_head).transpose(0, 1)
+
+        q_scaled = query_layer / self.norm_factor
+        if attn_mask is not None:
+            attention_probs = torch.baddbmm(attn_mask, q_scaled, key_layer.transpose(-2, -1))
+        else:
+            attention_probs = torch.bmm(q_scaled, key_layer.transpose(-2, -1))
+        attention_probs = attention_probs.softmax(dim=-1)
+
+        value_layer = value_layer.view(sk,
+            b * self.num_attention_heads_per_partition,
+            self.hidden_size_per_attention_head).transpose(0, 1)
+
+        # matmul: [b * np, sq, hn]
+        context_layer = torch.bmm(attention_probs, value_layer)
+
+        # change view [b, np, sq, hn]
+        context_layer = context_layer.view(b,
+            self.num_attention_heads_per_partition,
+            sq, self.hidden_size_per_attention_head)
+
+        # [b, np, sq, hn] --> [sq, b, np, hn]
+        context_layer = context_layer.permute(2, 0, 1, 3).contiguous()
+
+        # [sq, b, np, hn] --> [sq, b, hp]
+        new_context_layer_shape = context_layer.size()[:-2] + \
+            (self.hidden_size_per_partition,)
+        context_layer = context_layer.view(*new_context_layer_shape)
+
+        output = self.out_proj(context_layer)
+
+        return output
+
+
+class VisualAttentionBlock(nn.Module):
+    def __init__(
+            self,
+            d_model: int,
+            n_head: int,
+            mlp_ratio: float = 4.0,
+            act_layer: Callable = nn.GELU,
+            norm_layer: Callable = nn.LayerNorm,
+            is_cross_attention: bool = False,
+    ):
+        super().__init__()
+
+        self.ln_1 = norm_layer(d_model)
+        if is_cross_attention:
+            self.ln_1_kv = norm_layer(d_model)
+
+        self.ln_2 = norm_layer(d_model)
+        mlp_width = int(d_model * mlp_ratio)
+        self.attn = VisualAttention(d_model, n_head)
+        self.mlp = nn.Sequential(OrderedDict([
+            ("c_fc", nn.Linear(d_model, mlp_width)),
+            ("gelu", act_layer()),
+            ("c_proj", nn.Linear(mlp_width, d_model))
+        ]))
+
+    def attention(
+            self,
+            q_x: torch.Tensor,
+            k_x: Optional[torch.Tensor] = None,
+            v_x: Optional[torch.Tensor] = None,
+            attn_mask: Optional[torch.Tensor] = None,
+    ):
+        k_x = k_x if k_x is not None else q_x
+        v_x = v_x if v_x is not None else q_x
+
+        attn_mask = attn_mask.to(q_x.dtype) if attn_mask is not None else None
+        return self.attn(q_x, k_x, v_x, attn_mask=attn_mask)
+
+    def forward(
+            self,
+            q_x: torch.Tensor,
+            k_x: Optional[torch.Tensor] = None,
+            v_x: Optional[torch.Tensor] = None,
+            attn_mask: Optional[torch.Tensor] = None,
+    ):
+        k_x = self.ln_1_kv(k_x) if hasattr(self, "ln_1_kv") and k_x is not None else None
+        v_x = self.ln_1_kv(v_x) if hasattr(self, "ln_1_kv") and v_x is not None else None
+
+        x = q_x + self.attention(q_x=self.ln_1(q_x), k_x=k_x, v_x=v_x, attn_mask=attn_mask)
+        x = x + self.mlp(self.ln_2(x))
+        return x
+
+
+class TransformerBlock(nn.Module):
+    def __init__(
+            self,
+            width: int,
+            layers: int,
+            heads: int,
+            mlp_ratio: float = 4.0,
+            act_layer: Callable = nn.GELU,
+            norm_layer: Callable = nn.LayerNorm,
+    ):
+        super().__init__()
+        self.width = width
+        self.layers = layers
+
+        self.resblocks = nn.ModuleList([
+            VisualAttentionBlock(
+                width, heads, mlp_ratio, act_layer=act_layer, norm_layer=norm_layer)
+            for _ in range(layers)
+        ])
+
+    def get_cast_dtype(self) -> torch.dtype:
+        return self.resblocks[0].mlp.c_fc.weight.dtype
+
+    def get_cast_device(self) -> torch.device:
+        return self.resblocks[0].mlp.c_fc.weight.device
+
+    def forward(self, x: torch.Tensor, attn_mask: Optional[torch.Tensor] = None):
+        for r in self.resblocks:
+            x = r(x, attn_mask=attn_mask)
+        return x
+
+
+class VisionTransformer(nn.Module):
+
+    def __init__(
+            self,
+            image_size: int,
+            patch_size: int,
+            width: int,
+            layers: int,
+            heads: int,
+            mlp_ratio: float,
+            n_queries: int = 256,
+            output_dim: int = 512,
+            **kwargs
+    ):
+        super().__init__()
+        image_height, image_width = self.image_size = (image_size, image_size)
+        patch_height, patch_width = self.patch_size = (patch_size, patch_size)
+        self.grid_size = (image_height // patch_height, image_width // patch_width)
+        self.output_dim = output_dim
+
+        mean = (0.48145466, 0.4578275, 0.40821073)
+        std = (0.26862954, 0.26130258, 0.27577711)
+        self.image_transform = transforms.Compose([
+            transforms.Resize(
+                (image_size, image_size),
+                interpolation=InterpolationMode.BICUBIC
+            ),
+            transforms.ToTensor(),
+            transforms.Normalize(mean=mean, std=std),
+        ])
+
+        self.conv1 = nn.Conv2d(in_channels=3, out_channels=width, kernel_size=patch_size, stride=patch_size, bias=False)
+
+        # class embeddings and positional embeddings
+        scale = width ** -0.5
+        self.positional_embedding = nn.Parameter(scale * torch.randn(256, width))
+
+        norm_layer = partial(nn.LayerNorm, eps=1e-6)
+        act_layer = nn.GELU
+
+        self.ln_pre = norm_layer(width)
+        self.transformer = TransformerBlock(
+            width,
+            layers,
+            heads,
+            mlp_ratio,
+            act_layer=act_layer,
+            norm_layer=norm_layer,
+        )
+
+        self.attn_pool = Resampler(
+            grid_size=int(math.sqrt(n_queries)),
+            embed_dim=output_dim,
+            num_heads=output_dim // 128,
+            kv_dim=width,
+            norm_layer=norm_layer,
+        )
+        self.ln_post = norm_layer(output_dim)
+        self.proj = nn.Parameter((output_dim** -0.5) * torch.randn(output_dim, output_dim))
+
+    def forward(self, x: torch.Tensor):
+        x = x.to(
+            dtype=self.transformer.get_cast_dtype(),
+            device=self.transformer.get_cast_device(),
+        )
+        # to patches
+        x = self.conv1(x)  # shape = [*, width, grid, grid]
+        x = x.reshape(x.shape[0], x.shape[1], -1)  # shape = [*, width, grid ** 2]
+        x = x.permute(0, 2, 1)  # shape = [*, grid ** 2, width]
+
+        x = x + get_abs_pos(self.positional_embedding, x.size(1))
+
+        x = self.ln_pre(x)
+
+        x = x.permute(1, 0, 2)  # NLD -> LND
+        x = self.transformer(x)
+        x = x.permute(1, 0, 2)  # LND -> NLD
+
+        x = self.attn_pool(x)
+        x = self.ln_post(x)
+        x = x @ self.proj
+
+        return x
+
+    def encode(self, image_paths: List[str]):
+        images = []
+        for image_path in image_paths:
+            if image_path.startswith("http://") or image_path.startswith("https://"):
+                image = Image.open(requests.get(image_path, stream=True).raw)
+            else:
+                image = Image.open(image_path)
+            image = image.convert("RGB")
+            images.append(self.image_transform(image))
+        images = torch.stack(images, dim=0)
+        return self(images)

QwenViT/qwen_vit_G.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:398d0e75f09d958acdda01b8d685dbd0974abd4f4d404d0a0d9b638799d4d136
+size 3871440720

SEED-X-17B/README.md

@@ -0,0 +1,105 @@
+---
+license: other
+license_name: license-seed-x-17b
+license_link: LICENSE
+---
+
+# SEED-X
+[![arXiv](https://img.shields.io/badge/arXiv-2404.14396-b31b1b.svg)](https://arxiv.org/abs/2404.14396) [![Demo](https://img.shields.io/badge/Gradio-Demo-orange)](https://139a5c1d085953f17b.gradio.live/) 
+
+We introduce SEED-X, a unified and versatile foundation model, which can serve as various multimodal AI assistants **in the real world** after different instruction tuning, capable of responding to a variety of user needs through unifying **multi-granularity comprehension and generation**.
+
+All models and inference code are released! 
+
+## News
+**2024-04-22** :hugs: We release the [models](https://huggingface.co/AILab-CVC/SEED-X-17B) including the pre-trained foundation model **SEED-X**, the general instruction-tuned model **SEED-X-I**, the editing model **SEED-X-Edit**, and our de-tokenier, which can generate realistic images from ViT features (w/o or w/ a condition image).
+
+**2024-04-22** :hugs: We release an online [gradio demo](https://139a5c1d085953f17b.gradio.live/) of a general instruction-tuned model SEED-X-I. SEED-X-I can follow multimodal instruction (including images with dynamic resolutions) and make responses with images, texts and bounding boxes in multi-turn conversation. SEED-X-I **does not support image manipulation**. If you want to experience SEED-X-Edit for high-precision image editing, the inference code and model will be released soon.
+
+## TODOs
+- [x] Release the multimodal foundation model SEED-X.
+- [x] Release the instruction-tuned model SEED-X-Edit for high-precision image editing.
+- [x] Release 3.7M in-house image editing data.
+
+![image](https://github.com/AILab-CVC/SEED-X/blob/main/demos/teaser.jpg?raw=true)
+
+![image](https://github.com/AILab-CVC/SEED-X/blob/main/demos/case_example.jpg?raw=true)
+
+
+## Usage
+
+### Dependencies
+- Python >= 3.8 (Recommend to use [Anaconda](https://www.anaconda.com/download/#linux))
+- [PyTorch >=2.0.1](https://pytorch.org/)
+- NVIDIA GPU + [CUDA](https://developer.nvidia.com/cuda-downloads)
+
+### Installation
+Clone the repo and install dependent packages
+
+  ```bash
+  git clone https://github.com/AILab-CVC/SEED-X.git
+  cd SEED-X
+  pip install -r requirements.txt
+  ```
+
+### Model Weights
+We release the pretrained De-Tokenizer, the pre-trained foundation model **SEED-X**, the general instruction-tuned model **SEED-X-I**, the editing model **SEED-X-Edit** in in [SEED-X-17B Hugging Face](https://huggingface.co/AILab-CVC/SEED-X-17B).
+
+You can also download them separately as below,
+- Check the SEED-X de-tokenizer weights in [AILab-CVC/seed-x-17b-de-tokenizer](https://huggingface.co/AILab-CVC/seed-x-17b-de-tokenizer)
+- Check the pre-trained foundation model **SEED-X** weights in [AILab-CVC/seed-x-17b-pretrain](https://huggingface.co/AILab-CVC/seed-x-17b-pretrain)
+- Check the general instruction-tuned model **SEED-X-I** weights in [AILab-CVC/seed-x-17b-instruct](https://huggingface.co/AILab-CVC/seed-x-17b-instruct)
+- Check  the editing model **SEED-X-Edit** weights in [AILab-CVC/seed-x-17b-edit](https://huggingface.co/AILab-CVC/seed-x-17b-edit)
+  
+Please download the checkpoints and save them under the folder `./pretrained`. For example, `./pretrained/seed_x`.
+
+You also need to download [stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) and [Qwen-VL-Chat](https://huggingface.co/Qwen/Qwen-VL-Chat), and save them under the folder `./pretrained`. Please use the following script to extract the weights of visual encoder in Qwen-VL-Chat.
+```bash
+python3 src/tools/reload_qwen_vit.py
+```
+### Inference with SEED-X De-tokenizer
+```bash
+# For image reconstruction with ViT image features
+python3 src/inference/eval_seed_x_detokenizer.py
+# For image reconstruction with ViT image features and conditional image
+python3 src/inference/eval_seed_x_detokenizer_with_condition.py
+```
+
+### Inference with pre-trained model SEED-X
+```bash
+# For image comprehension and detection
+python3 src/inference/eval_img2text_seed_x.py
+# For image generation
+python3 src/inference/eval_text2img_seed_x.py
+```
+
+### Inference with the general instruction-tuned model SEED-X-I
+```bash
+# For image comprehension and detection
+python3 src/inference/eval_img2text_seed_x_i.py
+# For image generation
+python3 src/inference/eval_text2img_seed_x_i.py
+```
+
+### Inference with the editing model SEED-X-Edit
+```bash
+# For image editing
+python3 src/inference/eval_img2edit_seed_x_edit.py
+```
+
+## Citation
+If you find the work helpful, please consider citing:
+```bash
+@article{ge2024seed,
+  title={SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation},
+  author={Ge, Yuying and Zhao, Sijie and Zhu, Jinguo and Ge, Yixiao and Yi, Kun and Song, Lin and Li, Chen and Ding, Xiaohan and Shan, Ying},
+  journal={arXiv preprint arXiv:2404.14396},
+  year={2024}
+}
+```
+
+
+## License
+`SEED` is licensed under the Apache License Version 2.0 except for the third-party components listed in [License](License_Seed-X.txt). 
+
+During training SEED-X, we freeze the original parameters of LLaMA2 and optimize the LoRA module.

cvlm_llama2_tokenizer_100img_and_224loc_addpatch/added_tokens.json

@@ -0,0 +1,332 @@
+{
+  "</img>": 32101,
+  "</patch>": 32103,
+  "<box_end>": 32329,
+  "<box_start>": 32328,
+  "<img>": 32100,
+  "<img_00000>": 32000,
+  "<img_00001>": 32001,
+  "<img_00002>": 32002,
+  "<img_00003>": 32003,
+  "<img_00004>": 32004,
+  "<img_00005>": 32005,
+  "<img_00006>": 32006,
+  "<img_00007>": 32007,
+  "<img_00008>": 32008,
+  "<img_00009>": 32009,
+  "<img_00010>": 32010,
+  "<img_00011>": 32011,
+  "<img_00012>": 32012,
+  "<img_00013>": 32013,
+  "<img_00014>": 32014,
+  "<img_00015>": 32015,
+  "<img_00016>": 32016,
+  "<img_00017>": 32017,
+  "<img_00018>": 32018,
+  "<img_00019>": 32019,
+  "<img_00020>": 32020,
+  "<img_00021>": 32021,
+  "<img_00022>": 32022,
+  "<img_00023>": 32023,
+  "<img_00024>": 32024,
+  "<img_00025>": 32025,
+  "<img_00026>": 32026,
+  "<img_00027>": 32027,
+  "<img_00028>": 32028,
+  "<img_00029>": 32029,
+  "<img_00030>": 32030,
+  "<img_00031>": 32031,
+  "<img_00032>": 32032,
+  "<img_00033>": 32033,
+  "<img_00034>": 32034,
+  "<img_00035>": 32035,
+  "<img_00036>": 32036,
+  "<img_00037>": 32037,
+  "<img_00038>": 32038,
+  "<img_00039>": 32039,
+  "<img_00040>": 32040,
+  "<img_00041>": 32041,
+  "<img_00042>": 32042,
+  "<img_00043>": 32043,
+  "<img_00044>": 32044,
+  "<img_00045>": 32045,
+  "<img_00046>": 32046,
+  "<img_00047>": 32047,
+  "<img_00048>": 32048,
+  "<img_00049>": 32049,
+  "<img_00050>": 32050,
+  "<img_00051>": 32051,
+  "<img_00052>": 32052,
+  "<img_00053>": 32053,
+  "<img_00054>": 32054,
+  "<img_00055>": 32055,
+  "<img_00056>": 32056,
+  "<img_00057>": 32057,
+  "<img_00058>": 32058,
+  "<img_00059>": 32059,
+  "<img_00060>": 32060,
+  "<img_00061>": 32061,
+  "<img_00062>": 32062,
+  "<img_00063>": 32063,
+  "<img_00064>": 32064,
+  "<img_00065>": 32065,
+  "<img_00066>": 32066,
+  "<img_00067>": 32067,
+  "<img_00068>": 32068,
+  "<img_00069>": 32069,
+  "<img_00070>": 32070,
+  "<img_00071>": 32071,
+  "<img_00072>": 32072,
+  "<img_00073>": 32073,
+  "<img_00074>": 32074,
+  "<img_00075>": 32075,
+  "<img_00076>": 32076,
+  "<img_00077>": 32077,
+  "<img_00078>": 32078,
+  "<img_00079>": 32079,
+  "<img_00080>": 32080,
+  "<img_00081>": 32081,
+  "<img_00082>": 32082,
+  "<img_00083>": 32083,
+  "<img_00084>": 32084,
+  "<img_00085>": 32085,
+  "<img_00086>": 32086,
+  "<img_00087>": 32087,
+  "<img_00088>": 32088,
+  "<img_00089>": 32089,
+  "<img_00090>": 32090,
+  "<img_00091>": 32091,
+  "<img_00092>": 32092,
+  "<img_00093>": 32093,
+  "<img_00094>": 32094,
+  "<img_00095>": 32095,
+  "<img_00096>": 32096,
+  "<img_00097>": 32097,
+  "<img_00098>": 32098,
+  "<img_00099>": 32099,
+  "<loc-0>": 32104,
+  "<loc-100>": 32204,
+  "<loc-101>": 32205,
+  "<loc-102>": 32206,
+  "<loc-103>": 32207,
+  "<loc-104>": 32208,
+  "<loc-105>": 32209,
+  "<loc-106>": 32210,
+  "<loc-107>": 32211,
+  "<loc-108>": 32212,
+  "<loc-109>": 32213,
+  "<loc-10>": 32114,
+  "<loc-110>": 32214,
+  "<loc-111>": 32215,
+  "<loc-112>": 32216,
+  "<loc-113>": 32217,
+  "<loc-114>": 32218,
+  "<loc-115>": 32219,
+  "<loc-116>": 32220,
+  "<loc-117>": 32221,
+  "<loc-118>": 32222,
+  "<loc-119>": 32223,
+  "<loc-11>": 32115,
+  "<loc-120>": 32224,
+  "<loc-121>": 32225,
+  "<loc-122>": 32226,
+  "<loc-123>": 32227,
+  "<loc-124>": 32228,
+  "<loc-125>": 32229,
+  "<loc-126>": 32230,
+  "<loc-127>": 32231,
+  "<loc-128>": 32232,
+  "<loc-129>": 32233,
+  "<loc-12>": 32116,
+  "<loc-130>": 32234,
+  "<loc-131>": 32235,
+  "<loc-132>": 32236,
+  "<loc-133>": 32237,
+  "<loc-134>": 32238,
+  "<loc-135>": 32239,
+  "<loc-136>": 32240,
+  "<loc-137>": 32241,
+  "<loc-138>": 32242,
+  "<loc-139>": 32243,
+  "<loc-13>": 32117,
+  "<loc-140>": 32244,
+  "<loc-141>": 32245,
+  "<loc-142>": 32246,
+  "<loc-143>": 32247,
+  "<loc-144>": 32248,
+  "<loc-145>": 32249,
+  "<loc-146>": 32250,
+  "<loc-147>": 32251,
+  "<loc-148>": 32252,
+  "<loc-149>": 32253,
+  "<loc-14>": 32118,
+  "<loc-150>": 32254,
+  "<loc-151>": 32255,
+  "<loc-152>": 32256,
+  "<loc-153>": 32257,
+  "<loc-154>": 32258,
+  "<loc-155>": 32259,
+  "<loc-156>": 32260,
+  "<loc-157>": 32261,
+  "<loc-158>": 32262,
+  "<loc-159>": 32263,
+  "<loc-15>": 32119,
+  "<loc-160>": 32264,
+  "<loc-161>": 32265,
+  "<loc-162>": 32266,
+  "<loc-163>": 32267,
+  "<loc-164>": 32268,
+  "<loc-165>": 32269,
+  "<loc-166>": 32270,
+  "<loc-167>": 32271,
+  "<loc-168>": 32272,
+  "<loc-169>": 32273,
+  "<loc-16>": 32120,
+  "<loc-170>": 32274,
+  "<loc-171>": 32275,
+  "<loc-172>": 32276,
+  "<loc-173>": 32277,
+  "<loc-174>": 32278,
+  "<loc-175>": 32279,
+  "<loc-176>": 32280,
+  "<loc-177>": 32281,
+  "<loc-178>": 32282,
+  "<loc-179>": 32283,
+  "<loc-17>": 32121,
+  "<loc-180>": 32284,
+  "<loc-181>": 32285,
+  "<loc-182>": 32286,
+  "<loc-183>": 32287,
+  "<loc-184>": 32288,
+  "<loc-185>": 32289,
+  "<loc-186>": 32290,
+  "<loc-187>": 32291,
+  "<loc-188>": 32292,
+  "<loc-189>": 32293,
+  "<loc-18>": 32122,
+  "<loc-190>": 32294,
+  "<loc-191>": 32295,
+  "<loc-192>": 32296,
+  "<loc-193>": 32297,
+  "<loc-194>": 32298,
+  "<loc-195>": 32299,
+  "<loc-196>": 32300,
+  "<loc-197>": 32301,
+  "<loc-198>": 32302,
+  "<loc-199>": 32303,
+  "<loc-19>": 32123,
+  "<loc-1>": 32105,
+  "<loc-200>": 32304,
+  "<loc-201>": 32305,
+  "<loc-202>": 32306,
+  "<loc-203>": 32307,
+  "<loc-204>": 32308,
+  "<loc-205>": 32309,
+  "<loc-206>": 32310,
+  "<loc-207>": 32311,
+  "<loc-208>": 32312,
+  "<loc-209>": 32313,
+  "<loc-20>": 32124,
+  "<loc-210>": 32314,
+  "<loc-211>": 32315,
+  "<loc-212>": 32316,
+  "<loc-213>": 32317,
+  "<loc-214>": 32318,
+  "<loc-215>": 32319,
+  "<loc-216>": 32320,
+  "<loc-217>": 32321,
+  "<loc-218>": 32322,
+  "<loc-219>": 32323,
+  "<loc-21>": 32125,
+  "<loc-220>": 32324,
+  "<loc-221>": 32325,
+  "<loc-222>": 32326,
+  "<loc-223>": 32327,
+  "<loc-22>": 32126,
+  "<loc-23>": 32127,
+  "<loc-24>": 32128,
+  "<loc-25>": 32129,
+  "<loc-26>": 32130,
+  "<loc-27>": 32131,
+  "<loc-28>": 32132,
+  "<loc-29>": 32133,
+  "<loc-2>": 32106,
+  "<loc-30>": 32134,
+  "<loc-31>": 32135,
+  "<loc-32>": 32136,
+  "<loc-33>": 32137,
+  "<loc-34>": 32138,
+  "<loc-35>": 32139,
+  "<loc-36>": 32140,
+  "<loc-37>": 32141,
+  "<loc-38>": 32142,
+  "<loc-39>": 32143,
+  "<loc-3>": 32107,
+  "<loc-40>": 32144,
+  "<loc-41>": 32145,
+  "<loc-42>": 32146,
+  "<loc-43>": 32147,
+  "<loc-44>": 32148,
+  "<loc-45>": 32149,
+  "<loc-46>": 32150,
+  "<loc-47>": 32151,
+  "<loc-48>": 32152,
+  "<loc-49>": 32153,
+  "<loc-4>": 32108,
+  "<loc-50>": 32154,
+  "<loc-51>": 32155,
+  "<loc-52>": 32156,
+  "<loc-53>": 32157,
+  "<loc-54>": 32158,
+  "<loc-55>": 32159,
+  "<loc-56>": 32160,
+  "<loc-57>": 32161,
+  "<loc-58>": 32162,
+  "<loc-59>": 32163,
+  "<loc-5>": 32109,
+  "<loc-60>": 32164,
+  "<loc-61>": 32165,
+  "<loc-62>": 32166,
+  "<loc-63>": 32167,
+  "<loc-64>": 32168,
+  "<loc-65>": 32169,
+  "<loc-66>": 32170,
+  "<loc-67>": 32171,
+  "<loc-68>": 32172,
+  "<loc-69>": 32173,
+  "<loc-6>": 32110,
+  "<loc-70>": 32174,
+  "<loc-71>": 32175,
+  "<loc-72>": 32176,
+  "<loc-73>": 32177,
+  "<loc-74>": 32178,
+  "<loc-75>": 32179,
+  "<loc-76>": 32180,
+  "<loc-77>": 32181,
+  "<loc-78>": 32182,
+  "<loc-79>": 32183,
+  "<loc-7>": 32111,
+  "<loc-80>": 32184,
+  "<loc-81>": 32185,
+  "<loc-82>": 32186,
+  "<loc-83>": 32187,
+  "<loc-84>": 32188,
+  "<loc-85>": 32189,
+  "<loc-86>": 32190,
+  "<loc-87>": 32191,
+  "<loc-88>": 32192,
+  "<loc-89>": 32193,
+  "<loc-8>": 32112,
+  "<loc-90>": 32194,
+  "<loc-91>": 32195,
+  "<loc-92>": 32196,
+  "<loc-93>": 32197,
+  "<loc-94>": 32198,
+  "<loc-95>": 32199,
+  "<loc-96>": 32200,
+  "<loc-97>": 32201,
+  "<loc-98>": 32202,
+  "<loc-99>": 32203,
+  "<loc-9>": 32113,
+  "<patch>": 32102
+}

cvlm_llama2_tokenizer_100img_and_224loc_addpatch/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<unk>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

cvlm_llama2_tokenizer_100img_and_224loc_addpatch/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
+size 499723

cvlm_llama2_tokenizer_100img_and_224loc_addpatch/tokenizer_config.json

@@ -0,0 +1,34 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "bos_token": {
+    "__type": "AddedToken",
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "clean_up_tokenization_spaces": false,
+  "eos_token": {
+    "__type": "AddedToken",
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "legacy": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": null,
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": {
+    "__type": "AddedToken",
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

seed_detokenizer/first_stage/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a06a56bcfe123b5a01f91fde57999879060cb1155ebfea1b75861d14db7a0c72
+size 2427381181

seed_detokenizer/second_stage/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e631015b687053c3d0bc94e438f73f5e8b311d872d24cd2b02c0601903eb6a0e
+size 10515136357

seed_x/agent/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:defcb3adb638056a2119dea6226ddcaee5407c0aae47be81daa675105ab9cf8b
+size 860741046

seed_x/llm/config.json

@@ -0,0 +1,26 @@
+{
+  "_name_or_path": "/chat_sh/share_300719895/user/sijiezhao/model_hub/Llama-2-13b-chat-hf",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 13824,
+  "max_position_embeddings": 2048,
+  "model_type": "llama",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 40,
+  "num_key_value_heads": 40,
+  "pad_token_id": 0,
+  "pretraining_tp": 2,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.30.2",
+  "use_cache": true,
+  "vocab_size": 32330
+}

seed_x/llm/generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "pad_token_id": 0,
+  "temperature": 0.9,
+  "top_p": 0.6,
+  "transformers_version": "4.30.2"
+}

seed_x/llm/pytorch_model-00001-of-00006.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b3cae98c7c9cde9d3ca71b89f6b0fa122257402e91655c6a4ab9ebf658c9e1dd
+size 9963302283

seed_x/llm/pytorch_model-00002-of-00006.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c02c558ce9c5f029ae1cde04c9ce7798da1f875654b7e827a90609436010dd7
+size 9940856385

seed_x/llm/pytorch_model-00003-of-00006.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:62933151997f0c4eabb3fe24d765620a80354033f3459a02a4ede9b278c2b151
+size 9940856943

seed_x/llm/pytorch_model-00004-of-00006.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e9cc585aba8ec09b374c67cc594d435f8c382aa0bae241236ff782c6d58d991
+size 9867415289

seed_x/llm/pytorch_model-00005-of-00006.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:65a8e17cd863cf356d993d1c59df9a59f95f2e138944290e6c6aac427c1bd9dd
+size 9867456961

seed_x/llm/pytorch_model-00006-of-00006.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9d9345e45a28ad3e13ae1696ee37c141b450b1ebc1044400c2b84d9b503f8f68
+size 2497234607

seed_x/llm/pytorch_model.bin.index.json

@@ -0,0 +1,410 @@
+{
+  "metadata": {
+    "total_size": 52076984320
+  },
+  "weight_map": {
+    "lm_head.weight": "pytorch_model-00006-of-00006.bin",
+    "model.embed_tokens.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.0.input_layernorm.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.0.mlp.down_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.0.mlp.gate_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.0.mlp.up_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.0.post_attention_layernorm.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.0.self_attn.k_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.0.self_attn.o_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.0.self_attn.q_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.0.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00006.bin",
+    "model.layers.0.self_attn.v_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.1.input_layernorm.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.1.mlp.down_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.1.mlp.gate_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.1.mlp.up_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.1.post_attention_layernorm.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.1.self_attn.k_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.1.self_attn.o_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.1.self_attn.q_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.1.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00006.bin",
+    "model.layers.1.self_attn.v_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.10.input_layernorm.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.10.mlp.down_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.10.mlp.gate_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.10.mlp.up_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.10.post_attention_layernorm.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.10.self_attn.k_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.10.self_attn.o_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.10.self_attn.q_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.10.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00006.bin",
+    "model.layers.10.self_attn.v_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.11.input_layernorm.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.11.mlp.down_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.11.mlp.gate_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.11.mlp.up_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.11.post_attention_layernorm.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.11.self_attn.k_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.11.self_attn.o_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.11.self_attn.q_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.11.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00006.bin",
+    "model.layers.11.self_attn.v_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.12.input_layernorm.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.12.mlp.down_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.12.mlp.gate_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.12.mlp.up_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.12.post_attention_layernorm.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.12.self_attn.k_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.12.self_attn.o_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.12.self_attn.q_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.12.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00006.bin",
+    "model.layers.12.self_attn.v_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.13.input_layernorm.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.13.mlp.down_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.13.mlp.gate_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.13.mlp.up_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.13.post_attention_layernorm.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.13.self_attn.k_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.13.self_attn.o_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.13.self_attn.q_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.13.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00006.bin",
+    "model.layers.13.self_attn.v_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.14.input_layernorm.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.14.mlp.down_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.14.mlp.gate_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.14.mlp.up_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.14.post_attention_layernorm.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.14.self_attn.k_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.14.self_attn.o_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.14.self_attn.q_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.14.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00006.bin",
+    "model.layers.14.self_attn.v_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.15.input_layernorm.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.15.mlp.down_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.15.mlp.gate_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.15.mlp.up_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.15.post_attention_layernorm.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.15.self_attn.k_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.15.self_attn.o_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.15.self_attn.q_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.15.self_attn.rotary_emb.inv_freq": "pytorch_model-00003-of-00006.bin",
+    "model.layers.15.self_attn.v_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.16.input_layernorm.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.16.mlp.down_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.16.mlp.gate_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.16.mlp.up_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.16.post_attention_layernorm.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.16.self_attn.k_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.16.self_attn.o_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.16.self_attn.q_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.16.self_attn.rotary_emb.inv_freq": "pytorch_model-00003-of-00006.bin",
+    "model.layers.16.self_attn.v_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.17.input_layernorm.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.17.mlp.down_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.17.mlp.gate_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.17.mlp.up_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.17.post_attention_layernorm.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.17.self_attn.k_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.17.self_attn.o_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.17.self_attn.q_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.17.self_attn.rotary_emb.inv_freq": "pytorch_model-00003-of-00006.bin",
+    "model.layers.17.self_attn.v_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.18.input_layernorm.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.18.mlp.down_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.18.mlp.gate_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.18.mlp.up_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.18.post_attention_layernorm.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.18.self_attn.k_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.18.self_attn.o_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.18.self_attn.q_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.18.self_attn.rotary_emb.inv_freq": "pytorch_model-00003-of-00006.bin",
+    "model.layers.18.self_attn.v_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.19.input_layernorm.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.19.mlp.down_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.19.mlp.gate_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.19.mlp.up_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.19.post_attention_layernorm.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.19.self_attn.k_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.19.self_attn.o_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.19.self_attn.q_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.19.self_attn.rotary_emb.inv_freq": "pytorch_model-00003-of-00006.bin",
+    "model.layers.19.self_attn.v_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.2.input_layernorm.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.2.mlp.down_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.2.mlp.gate_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.2.mlp.up_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.2.post_attention_layernorm.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.2.self_attn.k_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.2.self_attn.o_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.2.self_attn.q_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.2.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00006.bin",
+    "model.layers.2.self_attn.v_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.20.input_layernorm.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.20.mlp.down_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.20.mlp.gate_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.20.mlp.up_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.20.post_attention_layernorm.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.20.self_attn.k_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.20.self_attn.o_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.20.self_attn.q_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.20.self_attn.rotary_emb.inv_freq": "pytorch_model-00003-of-00006.bin",
+    "model.layers.20.self_attn.v_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.21.input_layernorm.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.21.mlp.down_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.21.mlp.gate_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.21.mlp.up_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.21.post_attention_layernorm.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.21.self_attn.k_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.21.self_attn.o_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.21.self_attn.q_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.21.self_attn.rotary_emb.inv_freq": "pytorch_model-00003-of-00006.bin",
+    "model.layers.21.self_attn.v_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.22.input_layernorm.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.22.mlp.down_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.22.mlp.gate_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.22.mlp.up_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.22.post_attention_layernorm.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.22.self_attn.k_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.22.self_attn.o_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.22.self_attn.q_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.22.self_attn.rotary_emb.inv_freq": "pytorch_model-00003-of-00006.bin",
+    "model.layers.22.self_attn.v_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "model.layers.23.input_layernorm.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.23.mlp.down_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.23.mlp.gate_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.23.mlp.up_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.23.post_attention_layernorm.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.23.self_attn.k_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.23.self_attn.o_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.23.self_attn.q_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.23.self_attn.rotary_emb.inv_freq": "pytorch_model-00004-of-00006.bin",
+    "model.layers.23.self_attn.v_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.24.input_layernorm.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.24.mlp.down_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.24.mlp.gate_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.24.mlp.up_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.24.post_attention_layernorm.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.24.self_attn.k_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.24.self_attn.o_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.24.self_attn.q_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.24.self_attn.rotary_emb.inv_freq": "pytorch_model-00004-of-00006.bin",
+    "model.layers.24.self_attn.v_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.25.input_layernorm.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.25.mlp.down_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.25.mlp.gate_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.25.mlp.up_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.25.post_attention_layernorm.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.25.self_attn.k_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.25.self_attn.o_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.25.self_attn.q_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.25.self_attn.rotary_emb.inv_freq": "pytorch_model-00004-of-00006.bin",
+    "model.layers.25.self_attn.v_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.26.input_layernorm.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.26.mlp.down_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.26.mlp.gate_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.26.mlp.up_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.26.post_attention_layernorm.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.26.self_attn.k_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.26.self_attn.o_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.26.self_attn.q_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.26.self_attn.rotary_emb.inv_freq": "pytorch_model-00004-of-00006.bin",
+    "model.layers.26.self_attn.v_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.27.input_layernorm.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.27.mlp.down_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.27.mlp.gate_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.27.mlp.up_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.27.post_attention_layernorm.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.27.self_attn.k_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.27.self_attn.o_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.27.self_attn.q_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.27.self_attn.rotary_emb.inv_freq": "pytorch_model-00004-of-00006.bin",
+    "model.layers.27.self_attn.v_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.28.input_layernorm.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.28.mlp.down_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.28.mlp.gate_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.28.mlp.up_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.28.post_attention_layernorm.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.28.self_attn.k_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.28.self_attn.o_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.28.self_attn.q_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.28.self_attn.rotary_emb.inv_freq": "pytorch_model-00004-of-00006.bin",
+    "model.layers.28.self_attn.v_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.29.input_layernorm.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.29.mlp.down_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.29.mlp.gate_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.29.mlp.up_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.29.post_attention_layernorm.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.29.self_attn.k_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.29.self_attn.o_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.29.self_attn.q_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.29.self_attn.rotary_emb.inv_freq": "pytorch_model-00004-of-00006.bin",
+    "model.layers.29.self_attn.v_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.3.input_layernorm.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.3.mlp.down_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.3.mlp.gate_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.3.mlp.up_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.3.post_attention_layernorm.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.3.self_attn.k_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.3.self_attn.o_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.3.self_attn.q_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.3.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00006.bin",
+    "model.layers.3.self_attn.v_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.30.input_layernorm.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.30.mlp.down_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.30.mlp.gate_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.30.mlp.up_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.30.post_attention_layernorm.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.30.self_attn.k_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.30.self_attn.o_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.30.self_attn.q_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.30.self_attn.rotary_emb.inv_freq": "pytorch_model-00004-of-00006.bin",
+    "model.layers.30.self_attn.v_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "model.layers.31.input_layernorm.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.31.mlp.down_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.31.mlp.gate_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.31.mlp.up_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.31.post_attention_layernorm.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.31.self_attn.k_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.31.self_attn.o_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.31.self_attn.q_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.31.self_attn.rotary_emb.inv_freq": "pytorch_model-00005-of-00006.bin",
+    "model.layers.31.self_attn.v_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.32.input_layernorm.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.32.mlp.down_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.32.mlp.gate_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.32.mlp.up_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.32.post_attention_layernorm.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.32.self_attn.k_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.32.self_attn.o_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.32.self_attn.q_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.32.self_attn.rotary_emb.inv_freq": "pytorch_model-00005-of-00006.bin",
+    "model.layers.32.self_attn.v_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.33.input_layernorm.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.33.mlp.down_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.33.mlp.gate_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.33.mlp.up_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.33.post_attention_layernorm.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.33.self_attn.k_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.33.self_attn.o_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.33.self_attn.q_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.33.self_attn.rotary_emb.inv_freq": "pytorch_model-00005-of-00006.bin",
+    "model.layers.33.self_attn.v_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.34.input_layernorm.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.34.mlp.down_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.34.mlp.gate_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.34.mlp.up_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.34.post_attention_layernorm.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.34.self_attn.k_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.34.self_attn.o_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.34.self_attn.q_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.34.self_attn.rotary_emb.inv_freq": "pytorch_model-00005-of-00006.bin",
+    "model.layers.34.self_attn.v_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.35.input_layernorm.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.35.mlp.down_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.35.mlp.gate_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.35.mlp.up_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.35.post_attention_layernorm.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.35.self_attn.k_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.35.self_attn.o_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.35.self_attn.q_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.35.self_attn.rotary_emb.inv_freq": "pytorch_model-00005-of-00006.bin",
+    "model.layers.35.self_attn.v_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.36.input_layernorm.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.36.mlp.down_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.36.mlp.gate_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.36.mlp.up_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.36.post_attention_layernorm.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.36.self_attn.k_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.36.self_attn.o_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.36.self_attn.q_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.36.self_attn.rotary_emb.inv_freq": "pytorch_model-00005-of-00006.bin",
+    "model.layers.36.self_attn.v_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.37.input_layernorm.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.37.mlp.down_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.37.mlp.gate_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.37.mlp.up_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.37.post_attention_layernorm.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.37.self_attn.k_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.37.self_attn.o_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.37.self_attn.q_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.37.self_attn.rotary_emb.inv_freq": "pytorch_model-00005-of-00006.bin",
+    "model.layers.37.self_attn.v_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.38.input_layernorm.weight": "pytorch_model-00006-of-00006.bin",
+    "model.layers.38.mlp.down_proj.weight": "pytorch_model-00006-of-00006.bin",
+    "model.layers.38.mlp.gate_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.38.mlp.up_proj.weight": "pytorch_model-00006-of-00006.bin",
+    "model.layers.38.post_attention_layernorm.weight": "pytorch_model-00006-of-00006.bin",
+    "model.layers.38.self_attn.k_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.38.self_attn.o_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.38.self_attn.q_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.38.self_attn.rotary_emb.inv_freq": "pytorch_model-00005-of-00006.bin",
+    "model.layers.38.self_attn.v_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "model.layers.39.input_layernorm.weight": "pytorch_model-00006-of-00006.bin",
+    "model.layers.39.mlp.down_proj.weight": "pytorch_model-00006-of-00006.bin",
+    "model.layers.39.mlp.gate_proj.weight": "pytorch_model-00006-of-00006.bin",
+    "model.layers.39.mlp.up_proj.weight": "pytorch_model-00006-of-00006.bin",
+    "model.layers.39.post_attention_layernorm.weight": "pytorch_model-00006-of-00006.bin",
+    "model.layers.39.self_attn.k_proj.weight": "pytorch_model-00006-of-00006.bin",
+    "model.layers.39.self_attn.o_proj.weight": "pytorch_model-00006-of-00006.bin",
+    "model.layers.39.self_attn.q_proj.weight": "pytorch_model-00006-of-00006.bin",
+    "model.layers.39.self_attn.rotary_emb.inv_freq": "pytorch_model-00006-of-00006.bin",
+    "model.layers.39.self_attn.v_proj.weight": "pytorch_model-00006-of-00006.bin",
+    "model.layers.4.input_layernorm.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.4.mlp.down_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.4.mlp.gate_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.4.mlp.up_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.4.post_attention_layernorm.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.4.self_attn.k_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.4.self_attn.o_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.4.self_attn.q_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.4.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00006.bin",
+    "model.layers.4.self_attn.v_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.5.input_layernorm.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.5.mlp.down_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.5.mlp.gate_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.5.mlp.up_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.5.post_attention_layernorm.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.5.self_attn.k_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.5.self_attn.o_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.5.self_attn.q_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.5.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00006.bin",
+    "model.layers.5.self_attn.v_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.6.input_layernorm.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.6.mlp.down_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.6.mlp.gate_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.6.mlp.up_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.6.post_attention_layernorm.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.6.self_attn.k_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.6.self_attn.o_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.6.self_attn.q_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.6.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00006.bin",
+    "model.layers.6.self_attn.v_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.7.input_layernorm.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.7.mlp.down_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.7.mlp.gate_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.7.mlp.up_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.7.post_attention_layernorm.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.7.self_attn.k_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.7.self_attn.o_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.7.self_attn.q_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.7.self_attn.rotary_emb.inv_freq": "pytorch_model-00001-of-00006.bin",
+    "model.layers.7.self_attn.v_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "model.layers.8.input_layernorm.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.8.mlp.down_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.8.mlp.gate_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.8.mlp.up_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.8.post_attention_layernorm.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.8.self_attn.k_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.8.self_attn.o_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.8.self_attn.q_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.8.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00006.bin",
+    "model.layers.8.self_attn.v_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.9.input_layernorm.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.9.mlp.down_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.9.mlp.gate_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.9.mlp.up_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.9.post_attention_layernorm.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.9.self_attn.k_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.9.self_attn.o_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.9.self_attn.q_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.layers.9.self_attn.rotary_emb.inv_freq": "pytorch_model-00002-of-00006.bin",
+    "model.layers.9.self_attn.v_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "model.norm.weight": "pytorch_model-00006-of-00006.bin"
+  }
+}

seed_x_edit/agent/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a13dec9f7d43142ecfd2642f74cbf633f046ac6c7c5836c95e247dabb19ebd62
+size 860741046

seed_x_edit/llm/config.json

@@ -0,0 +1,26 @@
+{
+  "_name_or_path": "/chat_sh/share_300719895/user/yuyingge/jinguo_code/DiscreteLearning_debug/train_output/04_08_seedx_llama13b_vitg_448_anyres_pretrain/checkpoint-23000-merged/llm",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 13824,
+  "max_position_embeddings": 2048,
+  "model_type": "llama",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 40,
+  "num_key_value_heads": 40,
+  "pad_token_id": 0,
+  "pretraining_tp": 2,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.30.2",
+  "use_cache": true,
+  "vocab_size": 32330
+}

seed_x_edit/llm/generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "pad_token_id": 0,
+  "temperature": 0.9,
+  "top_p": 0.6,
+  "transformers_version": "4.30.2"
+}

seed_x_edit/llm/pytorch_model-00001-of-00006.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c264d2e8af73c29b05775089eb7cd2cea2725c90f31dd39672579e47a245e028
+size 9963302283

seed_x_edit/llm/pytorch_model-00002-of-00006.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:69d271bdd52398a44f24d76c664d4c60835f23defebdb72b5cc1f146dc90fc72
+size 9940856385

seed_x_edit/llm/pytorch_model-00003-of-00006.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:03e0d6671e3454d4b2e04c8913c1c1007ef3e6ffa9a516ed178c51ead0d0b8a6
+size 9940856943

seed_x_edit/llm/pytorch_model-00004-of-00006.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2eeac1037d3318fb095d95ff963c4f3277bd714b11d6b3a23962b354b7e7d335
+size 9867415289