SpringBoot切片上传+断点续传

概念

分片上传就是把一个大的文件分成若干块，一块一块的传输。这样做的好处可以减少重新上传的开销。比如：如果我们上传的文件是一个很大的文件，那么上传的时间应该会比较久，再加上网络不稳定各种因素的影响，很容易导致传输中断，用户除了重新上传文件外没有其他的办法，但是我们可以使用分片上传来解决这个问题。通过分片上传技术，如果网络传输中断，我们重新选择文件只需要传剩余的分片。而不需要重传整个文件，大大减少了重传的开销。

但是我们要如何选择一个合适的分片呢？因此我们要考虑如下几个事情：

1.分片越小，那么请求肯定越多，开销就越大。因此不能设置太小。

2.分片越大，灵活度就少了。

3.服务器端都会有个固定大小的接收Buffer。分片的大小最好是这个值的整数倍。

思路

1. 先对文件进行md5加密。使用md5加密的优点是：可以对文件进行唯一标识，同样可以为后台进行文件完整性校验进行比对。
2. 拿到md5值以后，服务器端查询下该文件是否已经上传过，如果已经上传过的话，就不用重新再上传。
3. 对大文件进行分片。比如一个100M的文件，我们一个分片是5M的话，那么这个文件可以分20次上传。
4. 向后台请求接口，接口里的数据就是我们已经上传过的文件块。(注意：为什么要发这个请求？就是为了能断点续传，比如我们使用百度网盘对吧，网盘里面有续传功能，当一个文件传到一半的时候，突然想下班不想上传了，那么服务器就应该记住我之前上传过的文件块，当我打开电脑重新上传的时候，那么它应该跳过我之前已经上传的文件块。再上传后续的块)。
5. 开始对未上传过的文件块进行上传。(这个是第二个请求，会把所有的分片合并，然后上传请求)。
6. 上传成功后，服务器会进行文件合并。最后完成。

大文件断点续传流程图

具体实现

前端代码

以vue为例

<template><div><input type="file" @change="handleFileChange"/><button @click="uploadFile">开始上传</button><progress :value="uploadProgress" max="100"></progress><span>{{ uploadProgress }}%</span></div>
</template><script setup>
import {ref} from 'vue';
import {progressFile, completeFile, chunkFile} from "@/api/file/file";
import SparkMD5 from 'spark-md5';
const {proxy} = getCurrentInstance();
const file = ref(null);
const uploadProgress = ref(0);
const chunkSize = 1024 * 1024 * 5; // 1MB
const uploadedChunks = ref([]);
const fileHash = ref('');
const fileName = ref('');
const handleFileChange = (event) => {file.value = event.target.files[0];if (file.value) {calculateFileHash();}
};const calculateFileHash = () => {return new Promise((resolve) => {const spark = new SparkMD5.ArrayBuffer();const reader = new FileReader();reader.onload = (e) => {spark.append(e.target.result);fileHash.value = spark.end();checkUploadProgress();resolve();};fileName.value = file.value.name;reader.readAsArrayBuffer(file.value);});
};const checkUploadProgress = async () => {try {progressFile({'fileHash': fileHash.value}).then((response) => {if (response.code === 200) {uploadedChunks.value = response.data;uploadProgress.value = Math.round((uploadedChunks.value.length / getTotalChunks()) * 100);}});} catch (error) {console.error('获取上传进度失败:', error);}
};const getTotalChunks = () => {return Math.ceil(file.value.size / chunkSize);
};const uploadFile = async () => {if (!file.value) return;const totalChunks = getTotalChunks();for (let i = 0; i < totalChunks; i++) {if (uploadedChunks.value.includes(i)) continue;const start = i * chunkSize;const end = Math.min(start + chunkSize, file.value.size);const chunk = file.value.slice(start, end);const formData = new FormData();formData.append('file', chunk);formData.append('fileHash', fileHash.value);formData.append('chunkIndex', i);formData.append('totalChunks', totalChunks);try {chunkFile(formData).then((response) => {if (response.code === 200) {uploadedChunks.value.push(i);localStorage.setItem(`upload-${fileHash.value}`, JSON.stringify(uploadedChunks.value));uploadProgress.value = Math.round((uploadedChunks.value.length / getTotalChunks()) * 100);if (uploadedChunks.value.length === totalChunks) {try {completeFile({fileHash: fileHash.value , fileName : fileName.value}).then((response) => {if (response.code === 200) {proxy.$modal.msgSuccess("上传成功！");}});} catch (error) {console.error('文件上传失败:', error);}}}});} catch (error) {console.error(`上传分片 ${i} 失败:`, error);break;}}
};
</script>

file.js封装

// 对axios的工具类
import request from '@/utils/request'// 判断文件是否存在
export function progressFile(query) {return request({url: '/file/progress',method: 'get',params: query})
}// 合并文件
export function completeFile(data) {return request({url: '/file/complete',  method: 'post',data: data})
}
// 切片文件上传
export function chunkFile(data) {return request({url: '/file/chunk',method: 'post',headers: {'Content-Type': 'application/x-www-form-urlencoded'},data: data})
}

后端代码

上传到服务器本地

具体实现方法

 private static final String UPLOAD_DIR = "D:/temp";@GetMapping("/progress")public AjaxResult getUploadProgress(@RequestParam String fileHash) {List<Integer> uploadedChunks = new ArrayList<>();File chunkDir = new File(UPLOAD_DIR + "/" + fileHash);if (chunkDir.exists() && chunkDir.isDirectory()) {for (File chunkFile : Objects.requireNonNull(chunkDir.listFiles())) {String filename = chunkFile.getName();if (filename.startsWith("part-")) {try {int chunkIndex = Integer.parseInt(filename.split("-")[1]);uploadedChunks.add(chunkIndex);} catch (NumberFormatException e) {// Ignore invalid chunk files}}}}return AjaxResult.success(uploadedChunks);}@PostMapping("/chunk")public AjaxResult uploadChunk(@RequestParam("file") MultipartFile file,@RequestParam String fileHash,@RequestParam int chunkIndex,@RequestParam int totalChunks) {try {// 验证文件哈希和分片索引if (fileHash == null || chunkIndex < 0 || chunkIndex >= totalChunks) {return AjaxResult.error();}// 创建临时存储目录Path tempDir = Paths.get(UPLOAD_DIR, fileHash);Files.createDirectories(tempDir);// 保存分片文件Path chunkPath = tempDir.resolve("part-" + chunkIndex);file.transferTo(chunkPath);return  AjaxResult.success();} catch (IOException e) {e.printStackTrace();return AjaxResult.error();}}@PostMapping("/complete")public AjaxResult completeUpload(@RequestBody Map<String, String> payload) {String fileHash = payload.get("fileHash");String fileName = payload.get("fileName");if (fileHash == null) {return AjaxResult.error();}try {Path tempDir = Paths.get(UPLOAD_DIR, fileHash);if (!tempDir.toFile().exists()) {return AjaxResult.error();}// 获取所有分片文件List<Path> chunks = new ArrayList<>();for (File chunkFile : Objects.requireNonNull(tempDir.toFile().listFiles())) {if (chunkFile.getName().startsWith("part-")) {chunks.add(chunkFile.toPath());}}// 按分片顺序排序chunks.sort((a, b) -> {int indexA = Integer.parseInt(a.getFileName().toString().split("-")[1]);int indexB = Integer.parseInt(b.getFileName().toString().split("-")[1]);return Integer.compare(indexA, indexB);});// 合并分片文件Path finalFilePath = Paths.get(UPLOAD_DIR, fileName);try (FileChannel outChannel = FileChannel.open(finalFilePath, StandardOpenOption.CREATE, StandardOpenOption.WRITE)) {for (Path chunkPath : chunks) {try (FileChannel inChannel = FileChannel.open(chunkPath, StandardOpenOption.READ)) {long position = outChannel.position();long count = inChannel.size();outChannel.transferFrom(inChannel, position, count);outChannel.position(position + count);}Files.delete(chunkPath); // 删除分片文件}}// 删除临时目录Files.delete(tempDir);return AjaxResult.success();} catch (IOException | NumberFormatException e) {e.printStackTrace();return AjaxResult.error();}}

返回工具类封装

public class AjaxResult extends HashMap<String, Object> {private static final long serialVersionUID = 1L;/*** 状态码*/public static final String CODE_TAG = "code";/*** 返回内容*/public static final String MSG_TAG = "msg";/*** 数据对象*/public static final String DATA_TAG = "data";/*** 初始化一个新创建的 AjaxResult 对象，使其表示一个空消息。*/public AjaxResult() {}/*** 初始化一个新创建的 AjaxResult 对象** @param code 状态码* @param msg  返回内容*/public AjaxResult(int code, String msg) {super.put(CODE_TAG, code);super.put(MSG_TAG, msg);}/*** 初始化一个新创建的 AjaxResult 对象** @param code 状态码* @param msg  返回内容* @param data 数据对象*/public AjaxResult(int code, String msg, Object data) {super.put(CODE_TAG, code);super.put(MSG_TAG, msg);if (data != null) {super.put(DATA_TAG, data);}}/*** 返回成功消息** @return 成功消息*/public static AjaxResult success() {return AjaxResult.success("操作成功");}/*** 返回成功数据** @return 成功消息*/public static AjaxResult success(Object data) {return AjaxResult.success("操作成功", data);}/*** 返回成功消息** @param msg 返回内容* @return 成功消息*/public static AjaxResult success(String msg) {return AjaxResult.success(msg, null);}/*** 返回成功消息** @param msg  返回内容* @param data 数据对象* @return 成功消息*/public static AjaxResult success(String msg, Object data) {return new AjaxResult(200, msg, data);}/*** 返回警告消息** @param msg 返回内容* @return 警告消息*/public static AjaxResult warn(String msg) {return AjaxResult.warn(msg, null);}/*** 返回警告消息** @param msg  返回内容* @param data 数据对象* @return 警告消息*/public static AjaxResult warn(String msg, Object data) {return new AjaxResult(601, msg, data);}/*** 返回错误消息** @return 错误消息*/public static AjaxResult error() {return AjaxResult.error("操作失败");}/*** 返回错误消息** @param msg 返回内容* @return 错误消息*/public static AjaxResult error(String msg) {return AjaxResult.error(msg, null);}/*** 返回错误消息** @param msg  返回内容* @param data 数据对象* @return 错误消息*/public static AjaxResult error(String msg, Object data) {return new AjaxResult(500, msg, data);}/*** 返回错误消息** @param code 状态码* @param msg  返回内容* @return 错误消息*/public static AjaxResult error(int code, String msg) {return new AjaxResult(code, msg, null);}/*** 是否为成功消息** @return 结果*/public boolean isSuccess() {return Objects.equals(200, this.get(CODE_TAG));}/*** 是否为警告消息** @return 结果*/public boolean isWarn() {return Objects.equals(601, this.get(CODE_TAG));}/*** 是否为错误消息** @return 结果*/public boolean isError() {return Objects.equals(500, this.get(CODE_TAG));}/*** 方便链式调用** @param key* @param value* @return*/@Overridepublic AjaxResult put(String key, Object value) {super.put(key, value);return this;}
}

上传到文件服务器

以minio服务器为例
在pom.xml文件中引入minio的依赖

<dependencies><!-- Spring Boot Web --><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-web</artifactId><version>3.3.4</version></dependency><!-- MinIO Java SDK --><dependency><groupId>io.minio</groupId><artifactId>minio</artifactId><version>8.5.7</version></dependency><!-- Lombok --><dependency><groupId>org.projectlombok</groupId><artifactId>lombok</artifactId><version>1.18.30</version><optional>true</optional></dependency>
</dependencies>

minio服务器配置FileUploadService.java

@Service
public class FileUploadService {@Value("${minio.endpoint:http://localhost:9991}")private String endpoint; // MinIO服务器地址@Value("${minio.access-key:root}")private String accessKey; // MinIO访问密钥@Value("${minio.secret-key:xxxx}")private String secretKey; // MinIO秘密密钥@Value("${minio.bucket-name:minio-xxxx}")private String bucketName; // 存储桶名称/*** 创建 MinIO 客户端** @return MinioClient 实例*/private MinioClient createMinioClient() {return MinioClient.builder().endpoint(endpoint).credentials(accessKey, secretKey).build();}/*** 如果存储桶不存在，则创建存储桶*/public void createBucketIfNotExists() throws IOException, NoSuchAlgorithmException, InvalidKeyException {MinioClient minioClient = createMinioClient();try {boolean found = minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucketName).build());if (!found) {minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucketName).build());}} catch (MinioException e) {throw new IOException("Error checking or creating bucket: " + e.getMessage(), e);}}/*** 上传文件分片到MinIO** @param fileId   文件标识符* @param filePart 文件分片* @return 分片对象名称*/public String uploadFilePart(String fileId, String fileName, MultipartFile filePart, Integer chunkIndex, Integer totalChunks) throws IOException, NoSuchAlgorithmException, InvalidKeyException {MinioClient minioClient = createMinioClient();try {// 构建分片对象名称String objectName = fileId + "/" + fileName + '-' + chunkIndex;// 设置上传参数PutObjectArgs putObjectArgs = PutObjectArgs.builder().bucket(bucketName).object(objectName).stream(filePart.getInputStream(), filePart.getSize(), -1).contentType(filePart.getContentType()).build();// 上传文件分片minioClient.putObject(putObjectArgs);return objectName;} catch (MinioException e) {throw new IOException("Error uploading file part: " + e.getMessage(), e);}}/*** 合并多个文件分片为一个完整文件*/public void mergeFileParts(FileMergeReqVO reqVO) throws IOException, NoSuchAlgorithmException, InvalidKeyException {MinioClient minioClient = createMinioClient();try {// 构建最终文件对象名称String finalObjectName = "merged/" + reqVO.getFileHash() + "/" + reqVO.getFileName();// 构建ComposeSource数组List<ComposeSource> sources = reqVO.getPartNames().stream().map(name ->ComposeSource.builder().bucket(bucketName).object(name).build()).toList();// 设置合并参数ComposeObjectArgs composeObjectArgs = ComposeObjectArgs.builder().bucket(bucketName).object(finalObjectName).sources(sources).build();// 合并文件分片minioClient.composeObject(composeObjectArgs);// 删除合并后的分片for (String partName : reqVO.getPartNames()) {minioClient.removeObject(RemoveObjectArgs.builder().bucket(bucketName).object(partName).build());}} catch (MinioException e) {throw new IOException("Error merging file parts: " + e.getMessage(), e);}}/*** 删除指定文件** @param fileName 文件名*/public void deleteFile(String fileName) throws IOException, NoSuchAlgorithmException, InvalidKeyException {MinioClient minioClient = createMinioClient();try {// 删除文件minioClient.removeObject(RemoveObjectArgs.builder().bucket(bucketName).object(fileName).build());} catch (MinioException e) {throw new IOException("Error deleting file: " + e.getMessage(), e);}}
}

FileMergeReqVO.java 实体

@Data
public class FileMergeReqVO {/*** 文件标识MD5*/private String fileHash;/*** 文件名*/private String fileName;/*** 合并文件列表*/@NotEmpty(message = "合并文件列表不允许为空")private List<String> partNames;
}

controller实现类

@AllArgsConstructor
@RestController
@RequestMapping("/file")
public class FileUploadController {private final FileUploadService fileUploadService;// 校验文件@GetMapping("/progress")public AjaxResult getUploadProgress(@RequestParam String fileHash) {List<Integer> uploadedChunks = new ArrayList<>();List<String> listFiles = new ArrayList<>();// 查询已经上传的文件分片// TODO 查询已上传分片文件信息// listFiles = for (File chunkFile : listFiles) {String filename = chunkFile.getName();if (filename.startsWith("part-")) {try {int chunkIndex = Integer.parseInt(filename.split("-")[1]);uploadedChunks.add(chunkIndex);} catch (NumberFormatException e) {// Ignore invalid chunk files}}}return AjaxResult.success(uploadedChunks);}/*** 上传文件分片** @param fileHash 文件标识符* @param file    文件分片* @param chunkIndex  当前分片索引* @param totalChunks 总分片数* @return 响应状态*/@PostMapping("/chunk")public AjaxResult<String> uploadFilePart(@PathVariable String fileHash,@RequestParam MultipartFile file,@RequestParam int chunkIndex,@RequestParam int totalChunks) {try {// 验证文件哈希和分片索引if (fileHash == null || chunkIndex < 0 || chunkIndex >= totalChunks) {return AjaxResult.error("上传失败！");}fileUploadService.createBucketIfNotExists();// 上传文件分片String objectName = fileUploadService.uploadFilePart(fileHash,"part-", file, chunkIndex);return AjaxResult.success("Uploaded file part: " + objectName);} catch (IOException | NoSuchAlgorithmException | InvalidKeyException e) {return AjaxResult.error(HttpStatus.INTERNAL_SERVER_ERROR.value(), "Error uploading file part: " + e.getMessage());}}/*** 合并文件分片** @param reqVO 参数* @return 响应状态*/@PostMapping("/complete")public AjaxResult<String> mergeFileParts(@RequestBody FileMergeReqVO reqVO) {try {// 查询切片列表List<String> chunks = new ArrayList<>();// TODO 查询全部切片列表// chunks  = ;// 按分片顺序排序chunks.sort((a, b) -> {int indexA = Integer.parseInt(a.split("-")[1]);int indexB = Integer.parseInt(b.split("-")[1]);return Integer.compare(indexA, indexB);});reqVO.setPartNames(chunks);fileUploadService.mergeFileParts(reqVO);return AjaxResult.success("File parts merged successfully.");} catch (IOException | NoSuchAlgorithmException | InvalidKeyException e) {return AjaxResult.error(HttpStatus.INTERNAL_SERVER_ERROR.value(), "Error merging file parts: " + e.getMessage());}}/*** 删除指定文件** @param fileId 文件ID* @return 响应状态*/@DeleteMapping("/delete/{fileHash}")public AjaxResult<String> deleteFile(@PathVariable String fileHash) {try {fileUploadService.deleteFile(fileHash);return AjaxResult.success("File deleted successfully.");} catch (IOException | NoSuchAlgorithmException | InvalidKeyException e) {return AjaxResult.error(HttpStatus.INTERNAL_SERVER_ERROR.value(), "Error deleting file: " + e.getMessage());}}
}

SpringBoot 设置文件传输大小限制

# spring配置
spring:servlet:multipart:max-file-size: 10MB     # 单个文件最大大小max-request-size: 50MB  # 整个请求的最大大小（例如多个文件总和）

如果设计nginx请求转发需进行如下配置

1. 全局设置

http {client_max_body_size 10M;  # 限制所有请求体的最大大小为10MB
}

2. 针对特定虚拟主机设置

server {listen 80;server_name example.com;client_max_body_size 20M;  # 限制该虚拟主机下的请求体最大为20MBlocation / {# 其他配置...}
}

3. 针对特定location设置

server {listen 80;server_name example.com;location /upload {client_max_body_size 5M;  # 仅限制/upload路径下的请求体最大为5MB}location / {# 其他配置...}
}