一、product-es准备

ES在内存中,所以在检索中优于mysql。ES也支持集群,数据分片存储。

需求:

  • 上架的商品才可以在网站展示。
  • 上架的商品需要可以被检索。

1.1 分析sku在es中如何存储

商品mapping

分析:商品上架在es中是存sku还是spu?

  • 1)、检索的时候输入名字,是需要按照sku的title进行全文检索的
  • 2)、检素使用商品规格,规格是spu的公共属性,每个spu是一样的
  • 3)、按照分类id进去的都是直接列出spu的,还可以切换。
  • 4〕、我们如果将sku的全量信息保存到es中(包括spu属性〕就太多字段了

方案1:

1
2
3
4
5
6
7
8
9
10
11
12
{
skuId:1
spuId:11
skyTitile:华为xx
price:999
saleCount:99
attr:[
{尺寸:5},
{CPU:高通945},
{分辨率:全高清}
]
缺点:如果每个sku都存储规格参数(如尺寸),会有冗余存储,因为每个spu对应的sku的规格参数都一样

方案2:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
sku索引
{
spuId:1
skuId:11
}
attr索引
{
skuId:11
attr:[
{尺寸:5},
{CPU:高通945},
{分辨率:全高清}
]
}
先找到4000个符合要求的spu,再根据4000个spu查询对应的属性,封装了4000个id,long 8B*4000=32000B=32KB
1K个人检索,就是32MB


结论:如果将规格参数单独建立索引,会出现检索时出现大量数据传输的问题,会引起网络网络

因此选用方案1,以空间换时间

1.2 建立product索引

最终选用的数据模型:

  • { “type”: “keyword” }, # 保持数据精度问题,可以检索,但不分词
  • “analyzer”: “ik_smart” # 中文分词器
  • “index”: false, # 不可被检索,不生成index
  • “doc_values”: false # 默认为true,不可被聚合,es就不会维护一些聚合的信息

视频原数据:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
PUT product
{
"mappings":{
"properties": {
"skuId":{
"type": "long"
},
"spuId":{
"type": "keyword"
},
"skuTitle": {
"type": "text",
"analyzer": "ik_smart"
},
"skuPrice": {
"type": "keyword"
},
"skuImg":{
"type": "keyword",
"index": false,
"doc_values": false
},
"saleCount":{
"type":"long"
},
"hasStock": {
"type": "boolean"
},
"hotScore": {
"type": "long"
},
"brandId": {
"type": "long"
},
"catalogId": {
"type": "long"
},
"brandName": {
"type": "keyword",
"index": false,
"doc_values": false
},
"brandImg":{
"type": "keyword",
"index": false,
"doc_values": false
},
"catalogName": {
"type": "keyword",
"index": false,
"doc_values": false
},
"attrs": {
"type": "nested",
"properties": {
"attrId": {
"type": "long"
},
"attrName": {
"type": "keyword",
"index": false,
"doc_values": false
},
"attrValue": {
"type": "keyword"
}
}
}
}
}
}

修改数据:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
PUT product
{
"mappings":{
"properties": {
"skuId":{ "type": "long" },
"spuId":{ "type": "keyword" }, # 不可分词
"skuTitle": {
"type": "text",
"analyzer": "ik_smart" # 中文分词器
},
"skuPrice": { "type": "keyword" }, # 保证精度问题
"skuImg" : { "type": "keyword" }, # 视频中有false
"saleCount":{ "type":"long" },
"hasStock": { "type": "boolean" },
"hotScore": { "type": "long" },
"brandId": { "type": "long" },
"catalogId": { "type": "long" },
"brandName": {"type": "keyword"}, # 视频中有false
"brandImg":{
"type": "keyword",
"index": false, # 不可被检索,不生成index,只用做页面使用
"doc_values": false # 不可被聚合,默认为true
},
"catalogName": {"type": "keyword" }, # 视频里有false
"attrs": {
"type": "nested",
"properties": {
"attrId": {"type": "long" },
"attrName": {
"type": "keyword",
"index": false,
"doc_values": false
},
"attrValue": {"type": "keyword" }
}
}
}
}
}

如果检索不到商品,自己用postman测试一下,可能有的字段需要更改,你也可以把没必要的”keyword”去掉

冗余存储的字段:不用来检索,也不用来分析,节省空间

库存是bool。

检索品牌id,但是不检索品牌名字、图片

用skuTitle检索

1.3 nested嵌入式对象

属性是”type”: “nested”,因为是内部的属性进行检索

数组类型的对象会被扁平化处理(对象的每个属性会分别存储到一起)

1
2
3
4
5
user.name=["aaa","bbb"]
user.addr=["ccc","ddd"]

这种存储方式,可能会发生如下错误:
错误检索到{aaa,ddd},这个组合是不存在的

数组的扁平化处理会使检索能检索到本身不存在的,为了解决这个问题,就采用了嵌入式属性,数组里是对象时用嵌入式属性(不是对象无需用嵌入式属性)

nested阅读:https://blog.csdn.net/weixin_40341116/article/details/80778599

使用聚合:https://blog.csdn.net/kabike/article/details/101460578

二、商品上架

按skuId上架

POST /product/spuinfo/{spuId}/up

1
2
3
4
5
6
@GetMapping("/skuId/{id}")
public R getSkuInfoBySkuId(@PathVariable("id") Long skuId){

SpuInfoEntity entity = spuInfoService.getSpuInfoBySkuId(skuId);
return R.ok().setData(entity);
}

product里组装好,search里上架

2.1 上架实体类

商品上架需要在es中保存spu信息并更新spu的状态信息,由于SpuInfoEntity与索引的数据模型并不对应,所以我们要建立专门的vo进行数据传输

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
@Data
public class SkuEsModel { //common中
private Long skuId;
private Long spuId;
private String skuTitle;
private BigDecimal skuPrice;
private String skuImg;
private Long saleCount;
private boolean hasStock;
private Long hotScore;
private Long brandId;
private Long catalogId;
private String brandName;
private String brandImg;
private String catalogName;
private List<Attr> attrs;

@Data
public static class Attr{
private Long attrId;
private String attrName;
private String attrValue;
}
}

2.2 库存量查询

上架要确保还有库存

  1. 在ware微服务里添加”查询sku是否有库存”的controller
1
2
3
4
5
6
7
8
9
10
11
// sku的规格参数相同,因此我们要将查询规格参数提前,只查询一次
/**
* 查询sku是否有库存
* 返回skuId 和 stock库存量
*/
@PostMapping("/hasStock")
public R getSkuHasStock(@RequestBody List<Long> SkuIds){
List<SkuHasStockVo> vos = wareSkuService.getSkuHasStock(SkuIds);
return R.ok().setData(vos);
}

  1. 然后用feign调用
  • WareFeignService.java
1
2
3
4
5
6
@FeignClient("gulimall-ware")
public interface WareFeignService {

@PostMapping("/ware/waresku/hasstock")
public R getSkuHasStock(@RequestBody List<Long> skuIds);
}
  • SearchFeignService.java
1
2
3
4
5
6
@FeignClient("gulimall-search")
public interface SearchFeignService {

@PostMapping(value = "/search/save//product")
public R productStatusUp(@RequestBody List<SkuEsModel> skuEsModels);
}
  1. 设置R的时候最后设置成泛型的
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
public class R extends HashMap<String, Object> {
private static final long serialVersionUID = 1L;

// 利用fastjson进行反序列化
public <T> T getData(TypeReference<T> typeReference){
Object data = get("data"); //默认是map
String jsonString = JSON.toJSONString(data);
T t = JSON.parseObject(jsonString,typeReference);
return t;
}

// private T data;
//
// public T getData() {
// return data;
// }
//
// public void setData(T data) {
// this.data = data;
// }
  1. 收集成map的时候,toMap()参数为两个方法,如SkyHasStockVo::getSkyId, item->item.getHasStock()

  2. 将封装好的SkuInfoEntity,调用search的feign,保存到es中

下面代码为更具sku的各种信息保存到es中

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
/**
* 上架商品
*/
@PostMapping("/product") // ElasticSaveController
public R productStatusUp(@RequestBody List<SkuEsModel> skuEsModels){

boolean status;
try {
status = productSaveService.productStatusUp(skuEsModels);
} catch (IOException e) {
log.error("ElasticSaveController商品上架错误: {}", e);
return R.error(BizCodeEnum.PRODUCT_UP_EXCEPTION.getCode(), BizCodeEnum.PRODUCT_UP_EXCEPTION.getMsg());
}
if(!status){
return R.ok();
}
return R.error(BizCodeEnum.PRODUCT_UP_EXCEPTION.getCode(), BizCodeEnum.PRODUCT_UP_EXCEPTION.getMsg());
}


public boolean productStatusUp(List<SkuEsModel> skuEsModels) throws IOException {
// 1.给ES建立一个索引 product
BulkRequest bulkRequest = new BulkRequest();
// 2.构造保存请求
for (SkuEsModel esModel : skuEsModels) {
// 设置索引
IndexRequest indexRequest = new IndexRequest(EsConstant.PRODUCT_INDEX);
// 设置索引id
indexRequest.id(esModel.getSkuId().toString());
String jsonString = JSON.toJSONString(esModel);
indexRequest.source(jsonString, XContentType.JSON);
// add
bulkRequest.add(indexRequest);
}
// bulk批量保存
BulkResponse bulk = client.bulk(bulkRequest, GuliESConfig.COMMON_OPTIONS);
// TODO 是否拥有错误
boolean hasFailures = bulk.hasFailures();
if(hasFailures){
List<String> collect = Arrays.stream(bulk.getItems()).map(item -> item.getId()).collect(Collectors.toList());
log.error("商品上架错误:{}",collect);
}
return hasFailures;
}
  1. 上架失败返回R.error(错误码,消息)

此时再定义一个错误码枚举。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
public enum BizCodeEnum {
UNKNOW_EXCEPTION(10000,"系统未知异常"),
VAILD_EXCEPTION(10001,"参数格式校验失败"),
PRODUCT_UP_EXCEPTION(11000,"商品上架异常");

private int code;
private String msg;
BizCodeEnum(int code, String msg){
this.code = code;
this.msg = msg;
}

public int getCode() {
return code;
}

public String getMsg() {
return msg;
}
}

在接收端获取他返回的状态码

6)上架后再让数据库中变为上架状态

7)mybatis为了能兼容接收null类型,要把long改为Long

debug时很容易远程调用异常,因为超时了

image-20210730224738884

2.3 根据spuId封装上架数据

前面我们写了把sku信息放到es中,但是这些信息需要我们封装,前端只是传过来了一个spuId

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
// SpuInfoServiceImpl 
public void upSpuForSearch(Long spuId) {
//1、查出当前spuId对应的所有sku信息,品牌的名字
List<SkuInfoEntity> skuInfoEntities=skuInfoService.getSkusBySpuId(spuId);
//TODO 4、根据spu查出当前sku的所有可以被用来检索的规格属性
List<ProductAttrValueEntity> productAttrValueEntities = productAttrValueService.list(new QueryWrapper<ProductAttrValueEntity>().eq("spu_id", spuId));
List<Long> attrIds = productAttrValueEntities.stream().map(attr -> {
return attr.getAttrId();
}).collect(Collectors.toList());
List<Long> searchIds=attrService.selectSearchAttrIds(attrIds);
Set<Long> ids = new HashSet<>(searchIds);
List<SkuEsModel.Attr> searchAttrs = productAttrValueEntities.stream().filter(entity -> {
return ids.contains(entity.getAttrId());
}).map(entity -> {
SkuEsModel.Attr attr = new SkuEsModel.Attr();
BeanUtils.copyProperties(entity, attr);
return attr;
}).collect(Collectors.toList());


//TODO 1、发送远程调用,库存系统查询是否有库存
Map<Long, Boolean> stockMap = null;
try {
List<Long> longList = skuInfoEntities.stream().map(SkuInfoEntity::getSkuId).collect(Collectors.toList());
List<SkuHasStockVo> skuHasStocks = wareFeignService.getSkuHasStocks(longList);
stockMap = skuHasStocks.stream().collect(Collectors.toMap(SkuHasStockVo::getSkuId, SkuHasStockVo::getHasStock));
}catch (Exception e){
log.error("远程调用库存服务失败,原因{}",e);
}

//2、封装每个sku的信息
Map<Long, Boolean> finalStockMap = stockMap;
List<SkuEsModel> skuEsModels = skuInfoEntities.stream().map(sku -> {
SkuEsModel skuEsModel = new SkuEsModel();
BeanUtils.copyProperties(sku, skuEsModel);
skuEsModel.setSkuPrice(sku.getPrice());
skuEsModel.setSkuImg(sku.getSkuDefaultImg());
//TODO 2、热度评分。0
skuEsModel.setHotScore(0L);
//TODO 3、查询品牌和分类的名字信息
BrandEntity brandEntity = brandService.getById(sku.getBrandId());
skuEsModel.setBrandName(brandEntity.getName());
skuEsModel.setBrandImg(brandEntity.getLogo());
CategoryEntity categoryEntity = categoryService.getById(sku.getCatalogId());
skuEsModel.setCatalogName(categoryEntity.getName());
//设置可搜索属性
skuEsModel.setAttrs(searchAttrs);
//设置是否有库存
skuEsModel.setHasStock(finalStockMap==null?false:finalStockMap.get(sku.getSkuId()));
return skuEsModel;
}).collect(Collectors.toList());


//TODO 5、将数据发给es进行保存:gulimall-search
R r = searchFeignService.saveProductAsIndices(skuEsModels);
if (r.getCode()==0){
this.baseMapper.upSpuStatus(spuId, ProductConstant.ProductStatusEnum.SPU_UP.getCode());
}else {
log.error("商品远程es保存失败");
}
}

2.4 测试

1
2
# 使用kibana测试
GET /product/_search

image-20210730225249064

三、商城系统首页

3.1 导入依赖

前端使用了thymeleaf开发,因此要导入该依赖,并且为了改动页面实时生效导入devtools

1
2
3
4
5
6
7
8
9
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
</dependency>

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-thymeleaf</artifactId>
</dependency>

可以在product项目中配置文件中配置

1
2
3
spring:
thymeleaf:
cache: false

image-20210730230335882

配置基础页面

image-20210730230546692

image-20210730230701488

测试:

image-20210730231001846

3.2 渲染一级分类菜单

由于访问首页时就要加载一级目录,所以我们需要在加载首页时获取该数据

1
2
3
4
5
6
7
@GetMapping({"/", "index.html"})
public String getIndex(Model model) {
//获取所有的一级分类
List<CategoryEntity> catagories = categoryService.getLevel1Catagories();
model.addAttribute("catagories", catagories);
return "index";
}
1
2
3
4
5
6
7
@Override
public List<CategoryEntity> getLevel1Catagories() {
// long start = System.currentTimeMillis();
List<CategoryEntity> parent_cid = this.list(new QueryWrapper<CategoryEntity>().eq("parent_cid", 0));
// System.out.println("查询一级菜单时间:"+(System.currentTimeMillis()-start));
return parent_cid;
}

页面遍历菜单数据

1
2
3
<li th:each="catagory:${catagories}" >
<a href="#" class="header_main_left_a" ctg-data="3" th:attr="ctg-data=${catagory.catId}"><b th:text="${catagory.name}"></b></a>
</li>

image-20210730231132280

3.3 渲染三级分类菜单

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
@GetMapping("index/catalog.json")
@ResponseBody
public Map<String, List<Catalog2Vo>> getCategoryMap() {
return categoryService.getCategoryMap();
}

public Map<String, List<Catalog2Vo>> getCategoryMap() {
List<CategoryEntity> categoryEntities = this.list(new QueryWrapper<CategoryEntity>().eq("cat_level", 2));

List<Catalog2Vo> catalog2Vos = categoryEntities.stream().map(categoryEntity -> {
List<CategoryEntity> level3 = this.list(new QueryWrapper<CategoryEntity>().eq("parent_cid", categoryEntity.getCatId()));
List<Catalog2Vo.Catalog3Vo> catalog3Vos = level3.stream().map(cat -> {
return new Catalog2Vo.Catalog3Vo(cat.getParentCid().toString(), cat.getCatId().toString(), cat.getName());
}).collect(Collectors.toList());
Catalog2Vo catalog2Vo = new Catalog2Vo(categoryEntity.getParentCid().toString(), categoryEntity.getCatId().toString(), categoryEntity.getName(), catalog3Vos);
return catalog2Vo;
}).collect(Collectors.toList());
Map<String, List<Catalog2Vo>> catalogMap = new HashMap<>();
for (Catalog2Vo catalog2Vo : catalog2Vos) {
List<Catalog2Vo> list = catalogMap.getOrDefault(catalog2Vo.getCatalog1Id(), new LinkedList<>());
list.add(catalog2Vo);
catalogMap.put(catalog2Vo.getCatalog1Id(),list);
}
return catalogMap;
}

image-20210731235248714

四、搭建域名访问环境

4.1 正向代理与反向代理

nginx就是通过反向代理实现负载均衡

4.2 Nginx配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
user  nginx;
worker_processes 1;

error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

#event块
events {
worker_connections 1024;
}

#http块
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;

log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

access_log /var/log/nginx/access.log main;

sendfile on;
#tcp_nopush on;

keepalive_timeout 65;

#gzip on;

upstream gulimall{
server 192.168.43.201:88;
}

include /etc/nginx/conf.d/*.conf;
############################################################################
#/etc/nginx/conf.d/default.conf 的server块
server {
listen 80;
server_name localhost;

#charset koi8-r;
#access_log /var/log/nginx/log/host.access.log main;

location / {
root /usr/share/nginx/html;
index index.html index.htm;
}

#error_page 404 /404.html;

# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}

# proxy the PHP scripts to Apache listening on 127.0.0.1:80
#
#location ~ \.php$ {
# proxy_pass http://127.0.0.1;
#}

# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
#location ~ \.php$ {
# root html;
# fastcgi_pass 127.0.0.1:9000;
# fastcgi_index index.php;
# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
# include fastcgi_params;
#}

# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# deny all;
#}
}

}

4.3 Nginx+Windows搭建域名访问环境

  1. 修改windows hosts文件改变本地域名映射,将gulimall.com映射到虚拟机ip

    image-20210801001551044

  2. 进入conf.d 文件,拷贝一份default.conf = >gulimall.conf

    image-20210801001918783

  3. 修改nginx的根配置文件nginx.conf,将upstream映射到我们的网关服务

1
2
3
upstream gulimall{
server 192.168.56.1:88;
}

image-20210801004341919

  1. 修改nginx的server块配置文件gulimall.conf,将以/开头的请求转发至我们配好的gulimallupstream,由于nginx的转发会丢失host头,所以我们添加头信息
1
2
3
4
location / {
proxy_pass http://gulimall;
proxy_set_header Host $host;
}

image-20210801003601190

image-20210801003519511

  1. 配置网关服务,将域名为**.gulimall.com转发至商品服务

    1
    2
    3
    4
    - id: gulimall_host
    uri: lb://gulimall-product
    predicates:
    - Host=**.gulimall.com